[MERGE] Sort knit fetches

[John Arbash Meinel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

John Arbash Meinel wrote:
> John Arbash Meinel wrote:
> ...

...

> I haven't timed this for 'bzr branch --stacked' yet, as I did the final
> work offline. But I'll run it when I get the chance.
> 
> John
> =:->

With this patch, the time for 'bzr branch --stacked' drops from 28m50s
down to 6m50s. Which isn't as fast as the 4m40s that I saw when
buffering everything into a single request.

My guess is that the pack repository is fairly fragmented, interweaving
the various texts. Which means that we get fewer collapsed regions, and
then run into the 200 limit for http Range requests.

Some of that I can tell, because via "bzr+ssh://" which has better
readv() handling, I get 3m30s for the 'all-at-once' and I get 3m49s for
the 5MB version.

In general, with the Range limitations, I see only 200k => 400k coming
back at a time, where I believe we set the max readv for the bzr
protocol to 5MB (since it doesn't support streaming).

I wonder if it would be good to set this max cache size to something
slightly less than the bzr+ssh size, to help avoid creating a 5MB
followed by a 100k request.

I also tried increasing the _bytes_to_read_before_seek from 128 to 1024
which dropped the time from 6m50s down to 6m5s.

With it set to 4096 the time went down to 5m43s.

Setting the _STREAM_MIN_BUFFER_SIZE up to 50MB, and the time dropped to
4m51s. (as expected it was buffered into a single request).

Anyway, the overall effect of this patch is a "good thing" for bzr
branch --stacked, and it seems to have a reasonable memory bound.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkmklCUACgkQJdeBCYSNAAOMlwCfVVfmJ8J6etLl2A35u93kFvZj
+SoAn08IzD+hwItdZ8F0RKposc06v2/V
=+ZPF
-----END PGP SIGNATURE-----