Excessive network traffic for pulling a small change

[John Arbash Meinel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Andrew Bennetts wrote:
> Stefan Monnier wrote:
>>> Although by contrast, doing the corresponding pull is a lot smaller via
>>> a smart protocol, and then doing the log is free.
>>> bzr -Dbytes -Dfetch pull -r 2251 \
>>>    bzr+ssh://bazaar.launchpad.net/~vcs-imports/grub/grub2-bzr
>>> Transferred: 286KiB (53.8K/s r:282K w:4K)
>> 286KB is still hard to justify for such a small change.
> 
> Is this a problem for you, or is it just that you were surprised?
> 
> I'll try to explain a little about why this particular smart server
> interaction took as much traffic as it did.
> 
> Basically, we make some tradeoffs to reduce round trips at the expense
> of sending more data than is strictly necessary to reduce wall
> clock time.  This expense is usually pretty insignificant, but when the
> actually data to transfer is small it can be larger — but 286kB is still
> not that much.
> 
> For example, the server deliberately pads out the response to
> “Repository.get_parent_map” to about 64kB, even if the request only
> asked about one item, because most of the time the client will need that
> information soon anyway, and an extra 60kB is rarely noticeable.  So
> that's a small part of the cost here (maybe 50kB).
> 
> Also, bzr doesn't just store deltas of file versions.  At regular
> intervals there will tend to be a fulltext version in the repository so
> that accessing arbitrary versions doesn't require reading and
> decompressing all of the history of that file.   (I don't immediately
> recall the precise details of this for the 2a format, so pardon any
> vagueness in that description.)
> 
> Perhaps surprisingly, when transferring over the network we don't
> transform those fulltext records to deltas... this is simpler and also
> saves the processing effort of decompressing on one side and
> recompressing on the other; instead the records read off the wire can be
> written directly to disk on the assumption that they are already
> reasonably well packed for general use.  I don't recall the details, but
> I think we found under (what we assume to be) typical conditions this is
> faster than decompressing/recompressing.
> 
> So in this case what seems to have happened is that the server has
> transferred a (compressed) fulltext of the Changelog, and that accounts
> for about 200kB of that transfer.
> 
> I'm not sure that this is worth worrying about, e.g. on my usual
> internet connection 200kB takes less than a second to receive.  Is this
> behaviour a problem for you?
> 
> -Andrew.

I was sort of staying out of this as a 'known bug' of the 2a format, but
since it wasn't clear in Andrew's head, I'll explain.

At this point, 2a compresses texts in 'blocks'. (something like .tar.bz2
vs .zip) The main trick for extraction speed is that we insert a
fulltext into the block, and then compress against that fulltext for the
next text. The 3rd text is compressed against the 1st fulltext, and the
2nd *delta*. The 4th against 1st fultext, 2nd delta, 3rd delta, etc. And
if we find a given text doesn't compress well enough, then we insert
another fulltext. So say the 5th text is a fulltext, then the 6th will
be compressed against (fulltext, delta, delta, delta, delta, fulltext).

Anyway, what this means is that all blocks start with at least 1 fulltext.

What that means for the current smart protocol, is that all fetches will
copy at least 1 fulltext, regardless of the delta size. So a 1 byte
change to a 1MB file will transmit 1MB of data, though after
autopack/pack it will probably get stored as a small handful of bytes
again. (While it is in its own pack, it is still stored as 1MB on disk.)

The current protocol is smart enough to shrink a block before
transmission, if you aren't going to use enough of it. In the above
case, if you request text 1, then we will chop off the bytes for texts
2-5, etc. If you requested just text 5, then we have a heuristic which
says "you only need X bytes out of Y, if X / Y < Z: create a new block".
(Which is the 'recompressing block on-the-fly' message that people
dislike in the logs. I dislike it because we are doing it far to often.
Mostly because of cases where we grab 2 texts from block A then 2 from
block B, then 2 from block A again.)

Git specifically addresses this with 'skinny packs' sent over the wire.
Which is that they will recompute the deltas for all texts against what
they believe the target has, and send just that.

We do have plans to do this, but it is a bit complicated in the current
layering. You have to get from the top layer which knows what revisions
to send and which ones not to send, down to the bottom layer which is
dealing in text keys, and then determine when you want to rebuild a
block, and when it is better to use it as-is. You also have to work out
on the other side, how to fill in those missing fulltexts.

All doable (without a disk format change, but *with* a network RPC
change), but not done yet. The specific design is such that 2a is known
to be less efficient for dumb requests than 1.9 (because you'll always
grab a full block, even if you aren't going to use all of it).

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkuiL/sACgkQJdeBCYSNAAP7lQCgzPG6yM6n4HckBpCr37bPxhqo
NM8Ani+CkKImAX5Ha4s1cl9+MCESmXxJ
=ls1p
-----END PGP SIGNATURE-----