[MERGE/RFC] Remote.readv() breaks up request

[John Arbash Meinel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Asmodehn Shade wrote:
> Thanks a lot for the help John.
> 
> For those who are experiencing the same problem, and as John advised me,
> try the sftp:// protocol.
> The protocol enforce the size of the chunks being transferred, so that
> enable us to workaround the problem as a user, even if it is slower than
> bzr+ssh://, at least it works ;-)
> 
> Thanks again !
> Waiting for the next version ;-)
> 
> Cheers,
> 
> --
> Alex
> 

...

> 2) Teach just the client to never make huge requests. This is easiest to
> implement, but I feel it isn't really the full answer.
> 

So the attached patch changes the RemoteTransport to break up readv()
requests into 5MB chunks. It isn't properly tested, other than having
Asmodehn run it on his repository (which means I know it works, but I
would like regression tests.)

We can certainly tune how we want to do this. I had it at 50MB for a
while, but Asmodehn noticed that it tended to be rather 'bursty', where
it would make a requests, spend ~4 minutes downloading that, and then
spin a lot, then wait another 4 min. At 5MB the wait is down to about
24s, which is certainly better.

To get better, I think we should just implement the streaming support,
as that lets us tick as the data comes in (I believe the 'smart' code
reads in 64kB chunks anyway).

So the big change here is similar to what I did for building the
WorkingTree. Just break requests into manageable chunks and go from there.

For repos like Asmodehn's, this is really a "make it work" step. As
otherwise it tries to buffer 20GB, which is just never going to work. I
haven't tested it a lot to measure impact on other operations. But with
a 5MB buffer, I would imagine the impact would be small.

So I think this workaround might be reasonable enough for now, I
certainly would rather see us be able to stream a readv back on the
server, and have the client stream the data in as well. (We do it just
fine over other transports like http/sftp.)

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkkIwesACgkQJdeBCYSNAAN3DACeIDt6GJ0eJfB9hCW21BCjnEsL
zEgAnAtxjLJdrJ83cQqsgqZYwJZB18qW
=S+mj
-----END PGP SIGNATURE-----
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: remote_readv_sections.patch
Url: https://lists.ubuntu.com/archives/bazaar/attachments/20081029/e803fa4c/attachment-0001.diff