Excessive network traffic - How to reproduce

[Andrew Bennetts]

Andrew Bennetts wrote:
[...]
> Sure enough, a quick hack that partly implements this policy brings the overall
> transfer down to 1798kB (46.5kB/s r:1773kB w:25kB).  This just optimising
> subsequent lookups within the same index type (i.e. multiple rix lookups).  I
> think we can do much better again by sharing that information between indicies
> (for optimising e.g. rix lookups followed by iix lookups).
> 
> The code is at <lp:~spiv/bzr/smarter-index-search>.

Update: by sharing ordering hints across index types I now get this result:

1034kB (50.1kB/s r:1022kB w:12kB)

Which is greatly improved from my starting point of 2424kB (50.2kB/s r:2395kB
w:30kB).  In terms of wall clock time that halves the time for me.

The code is a little hackish, but actually not too bad.  It seems to pass tests,
I'll polish it and put it up for review for merging soon.

So now the majority of the transfer is fetching data from packs.  As stated
before I'm a bit surprised that so much needs to be transferred, but that may
just be the natural result of fetching entire compression blocks for the client
to decompress just to parts it is interested in.

-Andrew.