[MERGE] Sort knit fetches

[John Arbash Meinel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

So I was doing some timing on "bzr branch --stacked", which is pretty
much identical to "bzr co --lightweight" (at the moment). And I ended up
finding this change.

This change makes it so that when extracting data from a pack file, we
first group the request by its index_memo. For all current formats, the
index_memo makes it group by pack file, and then order in forward I/O order.

This caused "bzr branch --stacked
http://people.ubuntu.org/~jameinel/python/trunk" to be cut in half
(9m40s => 4m40s), and make much better use of my network bandwidth.

This is because when you switch pack files during _iter_records_... it
makes a *new* readv() request.

It isn't the whole fix, as the rest of the fix was to change
"get_record_stream()" code to request all file content at once, which
has negative impact on memory. But this is a prerequisite which
shouldn't be controversial.

Also, I would expect we could get *even better* results, except we still
have a max of 200 Range entries per request. (At 400 Apache starts
giving errors, so we try to be conservative.) Of course, the other way
to help with that is to have "bzr pack" optimize the order of texts so
that we get more contiguous regions.

(Note that 'bzr branch --stacked bzr+ssh' connects approx 6 times... :(.)

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkmkWp0ACgkQJdeBCYSNAAPrigCgt4rTHzHaFH3ysEg8B/hi4w2f
LlgAn2rl98YvPj0YtdmDtE9pWw3/5KNo
=kqkj
-----END PGP SIGNATURE-----
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: sort_knit_fetch.patch
Url: https://lists.ubuntu.com/archives/bazaar/attachments/20090224/42babb28/attachment-0001.diff