prefetch still broken with readv and paramiko 1.6.1

[Robey Pointer]

On 26 Jul 2006, at 12:13, John Arbash Meinel wrote:

> The attached bundle is built off of your earlier work (2 revs ago),  
> when
> it was still fast. Basically, all it does is handle that the response
> may not be the same size as the request, and just keeps the queue  
> accurate.
>
> In my testing, it still works, even with oversized requests. Now, I
> don't know what sftp servers you have tried to download against.  
> And if
> they bork if you make a large request, or if they do what mine  
> does, and
> just return less data.
>
> It seems premature to restrict yourself to the lowest common  
> denominator
> in the request stage, if you can handle it properly in the response
> stage. However, in doing a bit more testing, it would seem that  
> servers
> only reply with whatever they consider max_bytes to be, and you  
> have to
> issue another request for the rest of the data. So I went ahead and
> updated my patch so that we can make short requests.
> It would be nice if we could use the 'Supported Features' response,
> which includes the max-read-size. But that may be a v6 versus v3  
> thing.

Yeah, I agree, handling shorter responses would be "right", but add  
enough latency to ruin the prefetching.  Thanks for going to the  
effort of trying it out, though.


> I think the reason that paramiko-trunk is so slow is because we have a
> fairly large queue at that point, and iterating through the queue  
> every
> time is slow. At least that is my guess. Something weird is going on,
> considering it effects seek+read performance without prefetch or readv
> enabled.
>
> I'm not positive, though.
>
> This are my results:
>
> paramiko-trunk readv	98.89
> paramiko-trunk seek+read	96.81
> request 32k readv	79.75
> request 32k seek+read	87.64
>
> Anyway, it might be possible to rewrite your trunk such that it is  
> still
> fast, and allows readv and prefetch to operate simultaneously. I'm not
> 100% positive as to how to do that, though.

This is bad.  Can you point me to the code you used to get those  
stats?  I'd like to dig in further and see if I can figure out what's  
going on in paramiko-land.  The queue management when using readv  
without prefetch ought to be basically the same as before...

robey