Excessive network traffic - How to reproduce

[Andrew Bennetts]

Andrew Bennetts wrote:
[...]
> What's interesting to me here is that we search 11 indices for each of
> revisions, texts, CHKs, inventories and signatures... but only end up reading
> from 2 pack files.  When searching indices we probably should be first searching
> indices for pack names that we have already found hits in from previous
> searches.  Quick estimate is that would save over 1MB of transfer, and about 40
> requests (which is a pretty large number of round-trips!).  I thought we already
> did that, but apparently not!

Sure enough, a quick hack that partly implements this policy brings the overall
transfer down to 1798kB (46.5kB/s r:1773kB w:25kB).  This just optimising
subsequent lookups within the same index type (i.e. multiple rix lookups).  I
think we can do much better again by sharing that information between indicies
(for optimising e.g. rix lookups followed by iix lookups).

The code is at <lp:~spiv/bzr/smarter-index-search>.

-Andrew.