Excessive network traffic - How to reproduce

[Andrew Bennetts]

Short answer: yes, packing the remote repository will significantly cut down on
the number of requests, and on the number of bytes bzr needs to read.  Bazaar is
searching 11 indices before finding the pack it needs.  That's just a temporary
workaround, and I think maybe bzr itself can do better relatively cheaply.  A
smart server is still the best solution by far, though.

More details below.

Colin D Bennett wrote:
[...]
> SUMMARY OF REPRODUCTION PROCESS
> 
> 1. Branch the remote GRUB trunk revision 2250 to a local standalone
> branch.
> 
> 2. Pull revision 2251 to the local branch.

Thanks, I can reproduce with that.  I see similar numbers: "2424kB (50.2kB/s
r:2395kB w:30kB)"

Here's a quick and dirty analysis of the HTTP requests by file-kind requested in
CSV format, sorted by bytes read:

Kind, Unique Files, Request Count, Total Bytes Read
pack,2,9,701266
cix,11,22,646033
tix,11,15,362406
rix,11,18,332695
iix,11,16,263580
six,11,11,31731
pack-names,1,1,507
format,3,3,488
branch-format,3,3,456
grub,1,1,411
conf,1,1,98
last-revision,1,1,57
tags,1,1,0

(The total bytes read here won't match the summary emitted by bzr exactly
because this is just counting HTTP response bodies, but it should be close
enough.)

What's interesting to me here is that we search 11 indices for each of
revisions, texts, CHKs, inventories and signatures... but only end up reading
from 2 pack files.  When searching indices we probably should be first searching
indices for pack names that we have already found hits in from previous
searches.  Quick estimate is that would save over 1MB of transfer, and about 40
requests (which is a pretty large number of round-trips!).  I thought we already
did that, but apparently not!

Separately, I'm a little surprised that such an apparently small change involves
transferring 700kB of actual pack data.  That seems quite high.

Here are more verbose details, by full path rather than summarised by kind,
sorted by most bytes read first.

Path, Request Count, Total Bytes Read
/r/grub/.bzr/repository/packs/1df3d09af711101db01567ab0360c457.pack,7,641429
/r/grub/.bzr/repository/indices/1f1a72546fe4af7958e87d664d316274.cix,5,212992
/r/grub/.bzr/repository/indices/9c860c2b7e382646b2018d5015a3f9d0.cix,5,176338
/r/grub/.bzr/repository/indices/e7f3204834edf9ac98cbfb031d6d345e.cix,4,143569
/r/grub/.bzr/repository/indices/1f1a72546fe4af7958e87d664d316274.rix,2,106364
/r/grub/.bzr/repository/indices/1f1a72546fe4af7958e87d664d316274.iix,2,106356
/r/grub/.bzr/repository/indices/9c860c2b7e382646b2018d5015a3f9d0.rix,5,80481
/r/grub/.bzr/repository/indices/9c860c2b7e382646b2018d5015a3f9d0.tix,2,78032
/r/grub/.bzr/repository/indices/ac7b7e6de9be8f8f40e641f63ff2f678.tix,2,78030
/r/grub/.bzr/repository/indices/e7f3204834edf9ac98cbfb031d6d345e.rix,2,74070
/r/grub/.bzr/repository/indices/e7f3204834edf9ac98cbfb031d6d345e.iix,2,73958
/r/grub/.bzr/repository/indices/e7f3204834edf9ac98cbfb031d6d345e.tix,2,73933
/r/grub/.bzr/repository/indices/1f1a72546fe4af7958e87d664d316274.tix,2,71350
/r/grub/.bzr/repository/indices/ac7b7e6de9be8f8f40e641f63ff2f678.rix,2,60466
/r/grub/.bzr/repository/packs/24d533aae38b7e4f04c965d62cfe667f.pack,2,59837
/r/grub/.bzr/repository/indices/ac7b7e6de9be8f8f40e641f63ff2f678.iix,2,59686
/r/grub/.bzr/repository/indices/ac7b7e6de9be8f8f40e641f63ff2f678.cix,1,35171
/r/grub/.bzr/repository/indices/7a627ee6f4f3326d5715a058d2c74375.cix,1,28752
/r/grub/.bzr/repository/indices/1df3d09af711101db01567ab0360c457.cix,1,25075
/r/grub/.bzr/repository/indices/e7f3204834edf9ac98cbfb031d6d345e.six,1,23353
/r/grub/.bzr/repository/indices/24d533aae38b7e4f04c965d62cfe667f.cix,1,22312
/r/grub/.bzr/repository/indices/7a627ee6f4f3326d5715a058d2c74375.tix,1,21420
/r/grub/.bzr/repository/indices/1df3d09af711101db01567ab0360c457.tix,1,19987
/r/grub/.bzr/repository/indices/24d533aae38b7e4f04c965d62cfe667f.tix,1,17518
/r/grub/.bzr/repository/indices/9c860c2b7e382646b2018d5015a3f9d0.iix,3,12288
/r/grub/.bzr/repository/indices/1df3d09af711101db01567ab0360c457.iix,1,3695
/r/grub/.bzr/repository/indices/1df3d09af711101db01567ab0360c457.rix,1,3694
/r/grub/.bzr/repository/indices/7a627ee6f4f3326d5715a058d2c74375.iix,1,3391
/r/grub/.bzr/repository/indices/7a627ee6f4f3326d5715a058d2c74375.rix,1,3388
/r/grub/.bzr/repository/indices/24d533aae38b7e4f04c965d62cfe667f.rix,1,3102
/r/grub/.bzr/repository/indices/24d533aae38b7e4f04c965d62cfe667f.iix,1,3076
/r/grub/.bzr/repository/indices/7a627ee6f4f3326d5715a058d2c74375.six,1,2595
/r/grub/.bzr/repository/indices/24d533aae38b7e4f04c965d62cfe667f.six,1,1806
/r/grub/.bzr/repository/indices/1df3d09af711101db01567ab0360c457.six,1,1798
/r/grub/.bzr/repository/indices/8f66816dd91abd01f0c8b3b4a9790e9e.tix,1,897
/r/grub/.bzr/repository/indices/8f66816dd91abd01f0c8b3b4a9790e9e.cix,1,807
/r/grub/.bzr/repository/indices/ac7b7e6de9be8f8f40e641f63ff2f678.six,1,676
/r/grub/.bzr/repository/indices/1f1a72546fe4af7958e87d664d316274.six,1,552
/r/grub/.bzr/repository/pack-names,1,507
/r/grub/.bzr/repository/indices/22badf0a00fcfbf32569a0aa05ddb5f8.tix,1,489
/r/grub/.bzr/repository/indices/49c9069da7c56a1c4e836730adab15ee.cix,1,458
/r/grub/.bzr/repository/indices/49c9069da7c56a1c4e836730adab15ee.tix,1,454
/r/grub/.bzr/repository/indices/8f66816dd91abd01f0c8b3b4a9790e9e.rix,1,433
/r/grub/.bzr/repository/indices/8f66816dd91abd01f0c8b3b4a9790e9e.iix,1,432
/r/grub/trunk/grub,1,411
/r/grub/trunk/grub/.bzr/repository/format,1,395
/r/grub/trunk/.bzr/branch-format,1,386
/r/grub/.bzr/repository/indices/4f628ab06c890d8befcd8b62df539e7a.cix,1,375
/r/grub/.bzr/repository/indices/4f628ab06c890d8befcd8b62df539e7a.tix,1,296
/r/grub/.bzr/repository/indices/8f66816dd91abd01f0c8b3b4a9790e9e.six,1,279
/r/grub/.bzr/repository/indices/49c9069da7c56a1c4e836730adab15ee.rix,1,259
/r/grub/.bzr/repository/indices/49c9069da7c56a1c4e836730adab15ee.iix,1,258
/r/grub/.bzr/repository/indices/49c9069da7c56a1c4e836730adab15ee.six,1,231
/r/grub/.bzr/repository/indices/22badf0a00fcfbf32569a0aa05ddb5f8.iix,1,230
/r/grub/.bzr/repository/indices/22badf0a00fcfbf32569a0aa05ddb5f8.rix,1,230
/r/grub/.bzr/repository/indices/4f628ab06c890d8befcd8b62df539e7a.iix,1,210
/r/grub/.bzr/repository/indices/4f628ab06c890d8befcd8b62df539e7a.rix,1,208
/r/grub/.bzr/repository/indices/22badf0a00fcfbf32569a0aa05ddb5f8.six,1,186
/r/grub/.bzr/repository/indices/22badf0a00fcfbf32569a0aa05ddb5f8.cix,1,184
/r/grub/.bzr/repository/indices/4f628ab06c890d8befcd8b62df539e7a.six,1,183
/r/grub/trunk/grub/.bzr/branch/branch.conf,1,98
/r/grub/.bzr/repository/indices/9c860c2b7e382646b2018d5015a3f9d0.six,1,72
/r/grub/trunk/grub/.bzr/branch/last-revision,1,57
/r/grub/.bzr/repository/format,1,54
/r/grub/trunk/grub/.bzr/branch/format,1,39
/r/grub/.bzr/branch-format,1,35
/r/grub/trunk/grub/.bzr/branch-format,1,35
/r/grub/trunk/grub/.bzr/branch/tags,1,0

-Andrew.