Paramiko throws EOFError rather than returning a 0 length file or ENOENT

[Vincent Ladeuil]

>>>>> "Robert" == Robert Collins <robertc at robertcollins.net> writes:

    Robert> On Thu, 2006-09-14 at 09:34 +0200, Vincent Ladeuil wrote:
    >> 
    >> The protocol gives a clear definition, yes. Implementing that
    >> clear EOF indication detection is a bit more complex than it
    >> appears but I've made good progress on that front* and I'm
    >> confident we can achieve that.

    Robert> Its a fairly straight forward statemachine: you are
    Robert> either reading the header of a chunk, or a chunk
    Robert> body. You exit the machine when you read 0 as the
    Robert> length for a valid chunk. (IIRC).

Oh yes, that part is easy enough and you remember correctly
(modulo the multipart handling). The problem is more in who
should do that and when, if the transport do it itself, we have
to hold the data in memory or save it to a temporary local file,
both solutions leads to other problems (who spoke about a 630M
file ?). If it's not the transport then it's outside of its
control and so is the verification of the socket state (which is
why I intend to re-integrate the http/response.py into the
transport framework but that will only partially address the
problem, if we let callers handle the file-like object tied to
the socket, what guaranty do we have that they will read all the
needed data before issuing a new request ? I know, I know, it
works so far so they do their duty, but the doubt remains :).

<snip/>

    Robert> Persistent connections are an optional feature in
    Robert> HTTP/1.0 and only spottily implemented.
    >> 
    >> Indeed, part of the tests I want to add is conformance to various
    >> levels of http protocol, again we can be conservative and just
    >> avoid persistent connections for 1.0 (as urllib2 already does
    >> even for 1.1...).
    >> 
    >> Also on that subject, I'd like to use the test suite against
    >> *real* servers and gather as much data as possible. It's a bit
    >> vague, I know, so any comments appreciated. That would help me
    >> un-fuzzy the strangeness :-)
    >> 
    Robert> I could go on :(
    >> 
    >> Please do ! Then I would just have to implement the tests instead
    >> of stressing myself about what *else* could go wrong :-)

    Robert> Well, consider error conditions - see the cherokee
    Robert> web server related bug reports for instance - where a
    Robert> partially conformant web server can interact very
    Robert> badly with bzr.

So far, what I have implemented in the bzr.urllib.keepalive
branch is effective but a bit "brutal" :

- if an error is encountered, close the connection and retry one
  time only (for one given request).

- consider all errors as ConnectionError.

That should provide a good user experience for any transient
errors while still giving up with a little delay for persistent
errors.

But I think I'm a bit overusing ConnectionError as meaning 'any
error related to a live connection' instead of 'we couldn't
connect to that server'. I intend to work on it these days but
get delayed by more work arriving on my plate.

    Robert> The basic crux of the issue is wide variety of
    Robert> implementations out there.

That's why I intend to work on the http test suite to ensure the
robustness of the http transports by implementing 'bogus' test
servers* allowing us to precisely test various parts of the
protocol (early closed connections, bogus or partial status line,
responses bodies when none are expected, no body when one is
expected, etc). May be I'm over-engineering here...

But on the other hand, I think more and more about re-using the
test suite to create a 'test_server' command that will determine
(sp?) if a server is sufficiently compliant to be used with
bzr.

The intent is to assist us debugging unknown servers, help users
configuring their servers**, etc.

    Vincent

* I'm not currently happy with the first test I wrote, I had to
  define *4* classes to implement 2 tests. What I think I need is
  a way to communicate between the test and the server to set
  options (close connection, send bogus status lines, etc). Any
  help on the design will be greatly appreciated (again see
  bzr.urllib.keepalive branch).

** Use case: bzr is so nice, let's set up a server (hpss, webdav,
   http, https, ftp, sftp, proxy, authentication,
   whatever). Let's fiddle the server conf. What's next ? How can
   I be sure my server configuration is correct ? Try some bzr
   commands ? Which ones ?  Etc, hope you get the idea.