sftp prefetch disabled

[Robey Pointer]

(combining 3 emails so I don't have to write 3 separate responses)


On 19 Jul 2006, at 11:44, John Arbash Meinel wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Robey Pointer wrote:
>>
>> On 19 Jul 2006, at 7:54, John Arbash Meinel wrote:
>>
>>> When I tested, it had no effect, but it turns out that was  
>>> because it
>>> wasn't actually enabled. :) (My paramiko is 1.5.2)
>>
>> Upgrade, slacker! :)
>>
>
> It is the one that comes with dapper. And 'debian sid' only comes with
> 1.5.0. Larstiq was just putting in a request to do an NMU (whatever  
> that
> means, but basically force an update to a package when the maintainer
> isn't doing their job).

Ugh.  I've been assuming they ship with whatever is current.  Is  
there anything I can do to help that happen?  Tell me whose kneecaps  
to break. ;)


>>> Does paramiko have an explicit readv() command? Right now we are  
>>> using a
>>> plain get followed by [seek + read(size), ...]
>>
>> Buh... Whoa.  I was sure Robert merged readv support in a few months
>> ago, but you're right, I don't see it.  What happened to that?
>
> He merged the ability to do readv using seek. I don't know of anything
> that calls out into paramiko for a direct readv() request.
>
> Even so, I think we don't want to use prefetch, because we don't want
> paramiko to cache all the file contents behind our back. My latest
> readv() work always makes sure to read in order, and buffer  
> overlapping
> requests, so the seek stuff wouldn't be an issue.
> But we don't want a 'bzr branch' of a large project over sftp to  
> have to
> cache all the file contents in ram.

Right, you don't want to use prefetch, but you do want to use readv.   
readv does prefetching over the requested chunks, yielding them as  
they arrive.  So if you're only going to do readv on a file, prefetch  
is going to be a waste of time.

At least this explains why your perf results surprised me. :)


On 19 Jul 2006, at 17:15, Robert Collins wrote:
>
> Your readv gathered all the data before returning it - I wanted one  
> that
> yields the data in the order and size requested - theres a mail thread
> or something where we discussed it.
>
> Last I heard, paramiko readv had not been changed to do that.

That started with 1.6.


> As Robey says - it was buggy. Specifically when we started doing
> emulated readv calls, prefetch on *all* paramikos at that time
> intereracted very badly. (By badly, I mean crash).
>
> Its on my todo to test bzr with 0.6.x paramiko and see if no warnings
> are emitted (last time I tried, it spewed awfully). Related to that
> LarstiQ has organised an update to the package to 0.6, and this is  
> part
> of that testing.

The problems with 1.6 were just an impedance mismatch, IIRC.  The  
last patch I sent was to correct for that by massaging the readv  
results into the form bzr wanted.  (bzr wants the offsets attached;  
paramiko just returns the data chunks.)

Unless I'm going senile, there was a branch for this, which seemed to  
be working after my patch.  I no longer have a copy of that branch,  
though, so if you don't either, then it's gone. :)  I guess it  
wouldn't be hard to do again, though.

robey