[MERGE] Re-order the packs during fetch

[John Arbash Meinel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Andrew Bennetts wrote:
> John Arbash Meinel wrote:
>> The attached patch changes the fetch logic slightly. After we have
>> determined/transmitted the revisions, we know what pack files those
>> revisions are in. Which means we also know what pack files the
>> inventories and texts are in. The existing code keeps looking in all
>> pack indexes for the inventories and texts.
>>
>> This patch just changes it, so that after we read revisions, we sort the
>> packs so that the interesting ones are put at the beginning of the list.
>>
>> In my testing, this helps a lot when we are transmitting just a small
>> amount of changes, that tend to be localized in certain packs. For
>> example, updating my bzr.1.9 branch with 1 new revision. The new code
>> still reads a lot of .rix info to find what revisions to transmit, but
>> then reads very little .iix and .tix because everything is in the first
>> pack it looks for.

One nice thing about this patch is that it helps both pack-0.92 repos as
well as --1.9 format branches. Here are some specific numbers, of how
long it takes me to update my local repository from bzr.dev 3874 to
bzr.dev 3876

37.2s  bzr1.9 from --format=pack-0.92
18.2s  bzr.dev from --format=pack-0.92
16.6s  bzr1.9 from --format=1.9
10.1s  bzr.dev from --format=1.9

So for the existing repositories, 'bzr pull' of a small amount of data
is almost 2x as fast, and for 1.9 format it is about 1.6:1. And that is
still 1.8:1 faster than 0.92 format. (So you get almost 4:1 upgrading
your repository to 1.9 and your client to 1.10rc1.)

And this is all done over plain HTTP with a latency around 200ms.

The next biggest win is probably the smart fetching that doesn't have to
probe the indexes directly. But there is also a big win in getting the
smart verbs for opening the branch and getting its information. It takes
3.2s (30%) before we have Branch.last_revision and are thinking about
fetching.

Getting streaming fetch will still be a big win. Of the 37kB we download
(from 1.9) only 5.5kB or so is actual .pack data.

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkk0W10ACgkQJdeBCYSNAAOBmwCbBX0a6Z8tRsqBs5AWYJKy4Qba
7UcAoNDBCPeaH6t4ShBE5xHYXVB4ajON
=MyGU
-----END PGP SIGNATURE-----