[RFC] Multipart support for _urllib_

[Martin Pool]

On 20 Jun 2006, John Arbash Meinel <john at arbash-meinel.com> wrote:

> > That is indeed impressive.  I wonder if we can get down towards that
> > number by just progressively replacing things in urllib and not
> > depending on pycurl?
> 
> I think pycurl gives us connection sharing/keep alive. Which becomes
> even more important over SSL.
> I'm a little curious if performance would be better if we shared the
> range-request object with the full-request object, but I have the
> feeling we tend to use either readv() or get(). We probably don't mix
> them around much.

Yes, connection sharing is important.  However, it should be possible to
do that within Python too, by just keeping a reference to the open
socket.  After the previous conversation about connection caching, we'd
probably want to share these among related transports (transports
created by cloning from each other.)

> Well, that shouldn't be too bad. How much testing do you want in place
> before it is considered worthy of merging? (I would like to get some
> testing as well, but I have some other major focuses this week, and
> would like to see it merged soonish).

Well, what I would *really* like is to encourage bzr developers to get
to into the habit of test-driven development; writing things as you go
along.  One thing I've learnt is that if you leave tests until later it
just gets harder to add them.

At least there should be a todo about what should be tested and perhaps
how it can be done (e.g. extending the http server).  You could also
probably fairly cheaply try that if you send a range requesto to the
SimpleHTTPServer the client does handle the response properly.

> Requests to readv are given in 'start, length', but ranges are from
> 'start, end'. It collapses overlapping/adjacent ranges (I could argue we
> could go one better and collapse if they are 'close enough').
> Probably an example makes it the most obvious (and we can add this to
> the doc string):
> 
> Examples:
>   [(5, 3)] => [(5, 8)]
>   [(5, 3), (6, 2)] = [(5,8)]
>   [(5, 3), (8, 2)] = [(5,8), (8,10)]
>   [(5, 3), (2, 1)] = [(2,3), (5,8)]
> 
> Would that be better?

Yep; in fact this is one case where I would say that at doctest is
actually a good way to document it (and incidentally test it).

-- 
Martin