prefetch still broken with readv and paramiko 1.6.1

[John Arbash Meinel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Robey Pointer wrote:
> 
> On 25 Jul 2006, at 9:28, John Arbash Meinel wrote:
> 
>> Well, I've found a few more bugs in the readv/prefetching logic.
> 
> Would you try the current bzr trunk?  Located here:
> 
>     http://www.lag.net/paramiko/bzr/paramiko
> 
> I took a 90 degree turn, decided to stop explaining how readv is meant
> to be used, and  tried to adapt it to how you're using it.  So I'd
> appreciate feedback on whether that makes it work right for you.
> 

It does work correctly, but unfortunately it makes everything much
slower. In testing regular seek+read, it slowed down operations from 80s
to ~100s.

...

> I still think that in general you won't want to use prefetch when you
> know you're just going to readv() a few sections of the file.  But that
> should at least work now and not be as huge a penalty as it was before.

I never really wanted to enable prefetch. I was just seeing that it did
help, which is why I wanted to switch to actually using readv().

The attached bundle is built off of your earlier work (2 revs ago), when
it was still fast. Basically, all it does is handle that the response
may not be the same size as the request, and just keeps the queue accurate.

In my testing, it still works, even with oversized requests. Now, I
don't know what sftp servers you have tried to download against. And if
they bork if you make a large request, or if they do what mine does, and
just return less data.

It seems premature to restrict yourself to the lowest common denominator
in the request stage, if you can handle it properly in the response
stage. However, in doing a bit more testing, it would seem that servers
only reply with whatever they consider max_bytes to be, and you have to
issue another request for the rest of the data. So I went ahead and
updated my patch so that we can make short requests.
It would be nice if we could use the 'Supported Features' response,
which includes the max-read-size. But that may be a v6 versus v3 thing.

I think the reason that paramiko-trunk is so slow is because we have a
fairly large queue at that point, and iterating through the queue every
time is slow. At least that is my guess. Something weird is going on,
considering it effects seek+read performance without prefetch or readv
enabled.

I'm not positive, though.

This are my results:

paramiko-trunk readv	98.89
paramiko-trunk seek+read	96.81
request 32k readv	79.75
request 32k seek+read	87.64

Anyway, it might be possible to rewrite your trunk such that it is still
fast, and allows readv and prefetch to operate simultaneously. I'm not
100% positive as to how to do that, though.

But the attached bundle works quite well in my testing.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.2 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFEx768JdeBCYSNAAMRApSQAJ9g++fX16OmsBFJzqa03q9r2ocGygCdFqzq
FRqNf/PuIfio/VBOmGwiAK0=
=YR35
-----END PGP SIGNATURE-----
-------------- next part --------------
A non-text attachment was scrubbed...
Name: paramiko-readv-response.patch
Type: text/x-patch
Size: 2287 bytes
Desc: not available
Url : https://lists.ubuntu.com/archives/bazaar/attachments/20060726/f247483c/attachment.bin