[MERGE] Updated sftp_readv

John Arbash Meinel john at arbash-meinel.com
Wed Dec 19 14:44:13 GMT 2007


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Vincent Ladeuil wrote:
>>>>>> "john" == John Arbash Meinel <john at arbash-meinel.com> writes:
> 
> <snip/>
> 
>     john> I'm also surprised to see that huge of a difference. If I'm counting correctly,
>     john> it makes it about 5x the total repository size. And I certainly don't see 5
>     john> copies in the code.
> 
>     john> 2x seems unavoidable (all = ''.join(data) has to have 2 copies), and I suppose
>     john> if there were multiple coalesced ranges and 'all' was still lying around that
>     john> would be 3 copies. And then splitting that data into its ranges would be a 4th
>     john> copy. I still don't see a 5th copy. I wonder if that is in higher level code.
> 
> I would "blame" paramiko instead (awaking, before coffee, guess ;),
> I think Vlad is using a sftp server on a local network and since
> _sftp_readv send all requests to paramiko at once before
> iterating the *results*, all sftp requests mat be fulfilled
> faster than _sftp_readv can consume them.
> 
> Threading the requests so that they can occur asynchronously is
> *good* for high latencies, but may be it should be limited a bit
> for high ones.
> 
>        Vincent
> 


I think you mean "should be limited a bit for *low* ones". But, as you say,
that is before coffee.

Anyway, that goes back to my idea for:

_max_ranges = 1000

for start in xrange(0, len(ranges), _max_ranges):
  for data in fp.readv(ranges[start:start+_max_ranges]):
    ...

As you noticed, we were sending 15k async requests, which seems a bit abusive.
This would limit it to 1000, at a time, which gives us 32KB*1000 = 32MB in
flight. Which sounds like enough to fill buffers. There will still be
1/1000*num_ranges more round trips than before, but it seems like a reasonable
way to cap the total number of requests on the wire.

Anyway, it would still be nice to get someone to review the patch :) (*wink*)
John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHaS49JdeBCYSNAAMRAsi6AKCnbJvCovXi3uU00gO9qoHGPFA3iQCeNpO7
MluS/D5ig2s8UniXN+CaRqk=
=as08
-----END PGP SIGNATURE-----



More information about the bazaar mailing list