Excessive network traffic - How to reproduce

[John Arbash Meinel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Andrew Bennetts wrote:
> Andrew Bennetts wrote:
> [...]
>> Sure enough, a quick hack that partly implements this policy brings the overall
>> transfer down to 1798kB (46.5kB/s r:1773kB w:25kB).  This just optimising
>> subsequent lookups within the same index type (i.e. multiple rix lookups).  I
>> think we can do much better again by sharing that information between indicies
>> (for optimising e.g. rix lookups followed by iix lookups).
>>
>> The code is at <lp:~spiv/bzr/smarter-index-search>.
> 
> Update: by sharing ordering hints across index types I now get this result:
> 
> 1034kB (50.1kB/s r:1022kB w:12kB)
> 
> Which is greatly improved from my starting point of 2424kB (50.2kB/s r:2395kB
> w:30kB).  In terms of wall clock time that halves the time for me.
> 
> The code is a little hackish, but actually not too bad.  It seems to pass tests,
> I'll polish it and put it up for review for merging soon.
> 
> So now the majority of the transfer is fetching data from packs.  As stated
> before I'm a bit surprised that so much needs to be transferred, but that may
> just be the natural result of fetching entire compression blocks for the client
> to decompress just to parts it is interested in.
> 
> -Andrew.
> 
> 
> 

I fixed this in Packer. It regressed in the StreamSource based code. I
believe because the layering was no longer clear how to do it, I didn't
get it restored, and because we were pushing to spend more time making
the smart server faster, and less time tweaking dumb fetches. (So it at
least got lower priority.)

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkubp7gACgkQJdeBCYSNAAOH3gCgo4C/FDaahbLtb7BQY5QuaZKD
G8MAn0f+JfC/uN0Bgh1vZV2QSNw4RyoJ
=VeIn
-----END PGP SIGNATURE-----