sftp prefetch disabled
Robey Pointer
robey at lag.net
Wed Jul 19 19:18:54 BST 2006
On 19 Jul 2006, at 7:54, John Arbash Meinel wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Robey Pointer wrote:
>>
>> There were bugs in the early implementations of prefetch. It
>> looks like
>> Robert just pushed the version requirement up; he didn't completely
>> disable it. As far as I can tell from the current code, prefetch is
>> enabled for all recent versions of paramiko.
>>
>> The version check could be changed to 1.6. There wasn't a 1.5.5 and
>> before 1.6 there were interaction issues if you used both readv() and
>> prefetch() (which bzr often tries to do, unfortunately). But the
>> version check should probably stay; I don't think it's safe to assume
>> that everyone has upgraded yet.
>
> All readv requests start out by doing a 'get()' and then seeking on
> the
> file. At one point I implemented a 'Transport._get_seekable' which
> defaulted to Transport.get, but sftp could override to provide an
> object
> that did not have prefetch enabled.
> When I tested, it had no effect, but it turns out that was because it
> wasn't actually enabled. :) (My paramiko is 1.5.2)
Upgrade, slacker! :)
> So we could enable prefetch for 1.5.2 on up, as long as we don't
> use it
> with readv(), right?
From memory, there were issues with it interacting with seek().
Originally prefetch() assumed you were going to read the file in
order, so if you asked for a later chunk, paramiko would throw away
everything up to that point. If you then asked for an earlier chunk,
it would trigger an assertion. Recent versions cope with this, and
actually hold on to chunks that you've skipped over, in case you're
going to come back.
There were also some edge cases, like it tripping out if you asked to
prefetch() when you were already at the end of the file.
>>> I will probably do a little bit of testing to see what effect
>>> prefetch
>>> has. (I thought I was, until I dug deep enough to see this). My gut
>>> feeling is that for readv() it would be a net loss, but for a plain
>>> 'get()' it would be good.
>>
>> I agree with that assessment. Currently readv() is implemented in
>> paramiko as a prefetch() over a limited range. That is, prefetch
>> () is
>> effectively readv(everything) without waiting for the results.
>
> Does paramiko have an explicit readv() command? Right now we are
> using a
> plain get followed by [seek + read(size), ...]
Buh... Whoa. I was sure Robert merged readv support in a few months
ago, but you're right, I don't see it. What happened to that?
robey
More information about the bazaar
mailing list