[MERGE] sftp readv performance tweaks (2x faster)
Robey Pointer
robey at lag.net
Sun Jul 23 06:29:51 BST 2006
On 20 Jul 2006, at 16:59, John Arbash Meinel wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Martin Pool wrote:
>> On 17 Jul 2006, John Arbash Meinel <john at arbash-meinel.com> wrote:
>>
>>> Well, I ran scripts all weekend to figure out how fast I could
>>> make sftp
>>> with my changes. And the attached image shows how it went.
>>
>> Very nice. Given the improvement it would be good for 0.9 - do
>> you have
>> any reservations?
>
> No specific reservations. I'm pretty happy with it overall. And we
> might
> be able to tweak a little performance improvement for local access
> as well.
>
> The only thing I really have left to test is whether we need to work
> around how paramiko does prefetch and/or readv. If paramiko directly
> implements readv() we might end up implementing a different
> SFTPTransport.readv(). I don't know how it is implemented though, and
> these changes should also benefit local requests and ftp requests.
Here's the short version:
def readv(self, chunks):
ordered_chunks = list(chunks)
ordered_chunks.sort(lambda x, y: cmp(x[0], y[0]))
self._start_prefetch(ordered_chunks)
# now we can just devolve to a bunch of read()s :)
for x in chunks:
self.seek(x[0])
yield self.read(x[1])
Maybe it can wait, though, until 1.6 is a requirement. I have a
branch where I ripped out pre-1.6 support just to see how much code
would be saved. It's a little out of date but I can revive it when
that's more viable.
(Actually I just merged it up to date in case you're curious, and
posted it here:
http://www.lag.net/~robey/code/bzr.dev.paramiko16
This should wait a while though; 1.6 isn't very old.)
robey
More information about the bazaar
mailing list