[MERGE] sftp readv performance tweaks (2x faster)

Robey Pointer robey at lag.net
Sun Jul 23 06:29:51 BST 2006


On 20 Jul 2006, at 16:59, John Arbash Meinel wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Martin Pool wrote:
>> On 17 Jul 2006, John Arbash Meinel <john at arbash-meinel.com> wrote:
>>
>>> Well, I ran scripts all weekend to figure out how fast I could  
>>> make sftp
>>> with my changes. And the attached image shows how it went.
>>
>> Very nice.  Given the improvement it would be good for 0.9 - do  
>> you have
>> any reservations?
>
> No specific reservations. I'm pretty happy with it overall. And we  
> might
> be able to tweak a little performance improvement for local access  
> as well.
>
> The only thing I really have left to test is whether we need to work
> around how paramiko does prefetch and/or readv. If paramiko directly
> implements readv() we might end up implementing a different
> SFTPTransport.readv(). I don't know how it is implemented though, and
> these changes should also benefit local requests and ftp requests.

Here's the short version:

     def readv(self, chunks):
         ordered_chunks = list(chunks)
         ordered_chunks.sort(lambda x, y: cmp(x[0], y[0]))
         self._start_prefetch(ordered_chunks)
         # now we can just devolve to a bunch of read()s :)
         for x in chunks:
             self.seek(x[0])
             yield self.read(x[1])

Maybe it can wait, though, until 1.6 is a requirement.  I have a  
branch where I ripped out pre-1.6 support just to see how much code  
would be saved.  It's a little out of date but I can revive it when  
that's more viable.

(Actually I just merged it up to date in case you're curious, and  
posted it here:
	http://www.lag.net/~robey/code/bzr.dev.paramiko16
This should wait a while though; 1.6 isn't very old.)

robey





More information about the bazaar mailing list