Why does the RemoteTransport readv reorder the ranges requested ?

[John Arbash Meinel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Robert Collins wrote:
> This seems counterproductive to me - by reordering we have to wait for
> data to arrive and buffer arbitrarily large amounts of it. Surely its
> better not to do that?
> 
> -Rob

In the general case, we request things in approximate order, so you don't have
to buffer much.

We already buffer the whole thing anyway, because the RPC calls are not
designed to incrementally read more data from the socket. (This should, of
course, be fixed, but at the moment we buffer everything in a StringIO anyway.)

When sending the request for different ranges, we probably want to collapse
locally as much as possible, so that we aren't sending 10,000 ranges across the
wire. I believe the collapse code reorders the request.
Arguably, we could have different collapse code that collapses as much as it
can without doing any reordering.

The other Transports do it that way.

We actually call "data = ...read_body_bytes()" which means that we actually
buffer everything in one big stream anyway. Though theoretically we could do
that incrementally. (If read_body_bytes() wasn't doing that itself anyway.)



So I think the simple answer is that it was expedient to do so. And the more
general answer is that we want to do at least some collapsing on the in-memory
end before sending the request out. For 'bzr branch REMOTE' means the
difference of sending 13,000 offset+length tuples over the wire, versus
probably sending ~10-100, depending on how fragmented the remote repository is.
(If it is completely unfragmented, then we may even send a single offset+length.)

Again, if we simply wrote new code to collapse the ranges, which didn't sort
them, we could get most of the benefit, and not have to buffer anything. (As
long as the read_body_bytes also didn't buffer everything.)

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFG1Jk4JdeBCYSNAAMRAgT0AKDKxrEbwvWvlhTWF3GB3sbEkGB50gCggZ37
qctWBg86sEbbBoSqQ2w5pB0=
=lsJk
-----END PGP SIGNATURE-----