[MERGE][bug #172701] Catch pycurl short reads and allows readv to issue several GET requests

Vincent Ladeuil v.ladeuil+lp at free.fr
Sat Dec 1 13:04:14 GMT 2007


>>>>> "john" == John Arbash Meinel <john at arbash-meinel.com> writes:

    john> Vincent Ladeuil wrote:
    john> ...

    >> No, I'm rewriting the http response parsing which should solve
    >> the problem in a better way. If *that* doesn't work I'll look
    >> into it, otherwise I prefer to minimize the requests issued.

    john> It is the wrong fix, yes. But it is the 30s fix.

Yes.

    john> I guess I can wait until you give me the uber-wonderful
    john> readv stream processing.

Under work. The results may turn to be wonderful but the code
will not be that complex. The main problem as we discussed about
on IRC is keeping the http socket clean and I'm now convinced
that the better way to do that is not to try hard at doing that
during the readv itself but instead before any new request. And
that may further simplify the whole handling of that pipe by
working around the one provided by httplib itself (which was not
primarily designed to handle persistent connections).

    >> 
    >> Also note that another solution, in the readv case, will be to
    >> try to issue all the GET requests and process the responses as
    >> they come back (httplib seems to be able to handle such a scheme,
    >> but that may be more work than just spawning a thread as paramiko).
    >> 
    >> Vincent
    >> 

    john> That sounds interesting. I would be worried that one
    john> would fail (for whatever reason) and then you would
    john> have a messed up queue.

Yes, the httplib is already is a bit brittle so going that way is
worrying and certainly not worth the effort since issuing the
requests in a thread will already gives us most of the benefits
we are aiming at in terms of latency

    john> Also, would that pipeline the request?

That's the idea yes.

    john> Would it properly reconnect for HTTP/1.0, and when you
    john> get a Connection: close response?

And these are the points to take into account ;-)

    john> If you can get it done, I'm very much looking forward
    john> to reviewing an incremental HTTP readv parser.

But that can be done without pipelining nor threading anything,
so I'll do that first.

   Vincent



More information about the bazaar mailing list