[MERGE] Updated sftp_readv
John Arbash Meinel
john at arbash-meinel.com
Wed Dec 19 14:44:13 GMT 2007
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Vincent Ladeuil wrote:
>>>>>> "john" == John Arbash Meinel <john at arbash-meinel.com> writes:
>
> <snip/>
>
> john> I'm also surprised to see that huge of a difference. If I'm counting correctly,
> john> it makes it about 5x the total repository size. And I certainly don't see 5
> john> copies in the code.
>
> john> 2x seems unavoidable (all = ''.join(data) has to have 2 copies), and I suppose
> john> if there were multiple coalesced ranges and 'all' was still lying around that
> john> would be 3 copies. And then splitting that data into its ranges would be a 4th
> john> copy. I still don't see a 5th copy. I wonder if that is in higher level code.
>
> I would "blame" paramiko instead (awaking, before coffee, guess ;),
> I think Vlad is using a sftp server on a local network and since
> _sftp_readv send all requests to paramiko at once before
> iterating the *results*, all sftp requests mat be fulfilled
> faster than _sftp_readv can consume them.
>
> Threading the requests so that they can occur asynchronously is
> *good* for high latencies, but may be it should be limited a bit
> for high ones.
>
> Vincent
>
I think you mean "should be limited a bit for *low* ones". But, as you say,
that is before coffee.
Anyway, that goes back to my idea for:
_max_ranges = 1000
for start in xrange(0, len(ranges), _max_ranges):
for data in fp.readv(ranges[start:start+_max_ranges]):
...
As you noticed, we were sending 15k async requests, which seems a bit abusive.
This would limit it to 1000, at a time, which gives us 32KB*1000 = 32MB in
flight. Which sounds like enough to fill buffers. There will still be
1/1000*num_ranges more round trips than before, but it seems like a reasonable
way to cap the total number of requests on the wire.
Anyway, it would still be nice to get someone to review the patch :) (*wink*)
John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFHaS49JdeBCYSNAAMRAsi6AKCnbJvCovXi3uU00gO9qoHGPFA3iQCeNpO7
MluS/D5ig2s8UniXN+CaRqk=
=as08
-----END PGP SIGNATURE-----
More information about the bazaar
mailing list