attn robey - paramiko readv [again]

Robert Collins robertc at robertcollins.net
Thu Apr 20 00:04:45 BST 2006


On Wed, 2006-04-19 at 15:27 -0700, Robey Pointer wrote:
> On 19 Apr 2006, at 6:18, Robert Collins wrote:
> 
> > Hi Robey,
> > 	I'm working on making knits the default in the test suite, and I'm
> > finding a set of lockups I can only put down to race conditions with
> > readv: if I step into the readv routine, it works, if I call it, it
> > locks up in recv in this thread.
> >
> > Would a newer paramiko (using 1.5.2 at the moment) help ? Or is it a
> > StubServer issue ?
> 
> You can't be using 1.5.2!  It didn't have readv. :)

We still have seek + read in the sftp transport code.

> Try 1.5.4 -- even though it was in 1.5.3 it wasn't quite working  
> right (and was undocumented).

bzr HEAD fixes the race I was reporting, but adds a new one:

  File
"/home/robertc/source/baz/integration/bzrlib/tests/test_permissions.py",
line 224, in test_new_files
    b_sftp.pull(b_local)
  File "/home/robertc/source/baz/integration/bzrlib/decorators.py", line
51, in write_locked
    return unbound(self, *args, **kwargs)
  File "/home/robertc/source/baz/integration/bzrlib/branch.py", line
1188, in pull
    return super(BzrBranch5, self).pull(source, overwrite,
stop_revision)
  File "/home/robertc/source/baz/integration/bzrlib/decorators.py", line
51, in write_locked
    return unbound(self, *args, **kwargs)
  File "/home/robertc/source/baz/integration/bzrlib/branch.py", line
1110, in pull
    old_count = len(self.revision_history())
  File "/home/robertc/source/baz/integration/bzrlib/decorators.py", line
38, in read_locked
    return unbound(self, *args, **kwargs)
  File "/home/robertc/source/baz/integration/bzrlib/branch.py", line
1024, in revision_history
    history = [l.rstrip('\r\n') for l in
  File "/home/robertc/source/baz/integration/bzrlib/decorators.py", line
38, in read_locked
    return unbound(self, *args, **kwargs)
  File "/home/robertc/source/baz/integration/bzrlib/lockable_files.py",
line 176, in get_utf8
    return codecs.getreader('utf-8')(self._transport.get(relpath))
  File "/home/robertc/source/baz/integration/bzrlib/transport/sftp.py",
line 382, in get
    f.prefetch()
  File "/home/robertc/source/baz/integration/paramiko/sftp_file.py",
line 365, in prefetch
    self._start_prefetch(chunks)
  File "/home/robertc/source/baz/integration/paramiko/sftp_file.py",
line 408, in _start_prefetch
    self._prefetch_so_far = chunks[0][0]
IndexError: list index out of range

I'll try using readv directly in paramiko now.

> I've made one tweak since then that's only in the bzr branch*, but it  
> would only affect rapidly doing multiple readv's on the same file, or  
> a prefetch followed immediately by a readv (readv is implemented as a  
> form of prefetch).  Is it possible that's what's happening?  If so,  
> the bzr branch should fix it.
> 
> [But if you do a prefetch followed-by readv, you will get terrible  
> performance anyway.  Maybe I should have readv notice that situation  
> and just use the prefetched data in that case.  Hmm.]
> 
> I'll need to make a 1.5.5 soon anyway, because Alexander helped me  
> find a bunch of Windows issues in the sftp unit tests.

Cool. We'll want to get a stable combination in dapper :).

Rob

-- 
GPG key available at: <http://www.robertcollins.net/keys.txt>.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 191 bytes
Desc: This is a digitally signed message part
Url : https://lists.ubuntu.com/archives/bazaar/attachments/20060420/83ed4b24/attachment.pgp 


More information about the bazaar mailing list