sftp prefetch disabled
John Arbash Meinel
john at arbash-meinel.com
Wed Jul 19 19:44:37 BST 2006
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Robey Pointer wrote:
>
> On 19 Jul 2006, at 7:54, John Arbash Meinel wrote:
>
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Robey Pointer wrote:
>>>
>>> There were bugs in the early implementations of prefetch. It looks like
>>> Robert just pushed the version requirement up; he didn't completely
>>> disable it. As far as I can tell from the current code, prefetch is
>>> enabled for all recent versions of paramiko.
>>>
>>> The version check could be changed to 1.6. There wasn't a 1.5.5 and
>>> before 1.6 there were interaction issues if you used both readv() and
>>> prefetch() (which bzr often tries to do, unfortunately). But the
>>> version check should probably stay; I don't think it's safe to assume
>>> that everyone has upgraded yet.
>>
>> All readv requests start out by doing a 'get()' and then seeking on the
>> file. At one point I implemented a 'Transport._get_seekable' which
>> defaulted to Transport.get, but sftp could override to provide an object
>> that did not have prefetch enabled.
>> When I tested, it had no effect, but it turns out that was because it
>> wasn't actually enabled. :) (My paramiko is 1.5.2)
>
> Upgrade, slacker! :)
>
It is the one that comes with dapper. And 'debian sid' only comes with
1.5.0. Larstiq was just putting in a request to do an NMU (whatever that
means, but basically force an update to a package when the maintainer
isn't doing their job).
>
>> So we could enable prefetch for 1.5.2 on up, as long as we don't use it
>> with readv(), right?
>
> From memory, there were issues with it interacting with seek().
> Originally prefetch() assumed you were going to read the file in order,
> so if you asked for a later chunk, paramiko would throw away everything
> up to that point. If you then asked for an earlier chunk, it would
> trigger an assertion. Recent versions cope with this, and actually hold
> on to chunks that you've skipped over, in case you're going to come back.
>
> There were also some edge cases, like it tripping out if you asked to
> prefetch() when you were already at the end of the file.
>
>
>>>> I will probably do a little bit of testing to see what effect prefetch
>>>> has. (I thought I was, until I dug deep enough to see this). My gut
>>>> feeling is that for readv() it would be a net loss, but for a plain
>>>> 'get()' it would be good.
>>>
>>> I agree with that assessment. Currently readv() is implemented in
>>> paramiko as a prefetch() over a limited range. That is, prefetch() is
>>> effectively readv(everything) without waiting for the results.
>>
>> Does paramiko have an explicit readv() command? Right now we are using a
>> plain get followed by [seek + read(size), ...]
>
> Buh... Whoa. I was sure Robert merged readv support in a few months
> ago, but you're right, I don't see it. What happened to that?
>
> robey
>
>
He merged the ability to do readv using seek. I don't know of anything
that calls out into paramiko for a direct readv() request.
Even so, I think we don't want to use prefetch, because we don't want
paramiko to cache all the file contents behind our back. My latest
readv() work always makes sure to read in order, and buffer overlapping
requests, so the seek stuff wouldn't be an issue.
But we don't want a 'bzr branch' of a large project over sftp to have to
cache all the file contents in ram.
John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.2 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFEvn2VJdeBCYSNAAMRAlIZAKCCgcLzhrYq6gEJRf6uEG7jsmUCewCgkOXe
iXNfea60/JvtmaQwDjrVJPU=
=T6b6
-----END PGP SIGNATURE-----
More information about the bazaar
mailing list