[merge] http multirange support

[John Arbash Meinel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

John Arbash Meinel wrote:
> The attached diff is an implementation of multi range support for http
> requests. It enables them on both http and urllib. And then goes one
> step further to re-use the curl connections to enable keep-alive support.
> 
> Thanks go to Michael Ellerman for writing the multi-range support, and
> Johan Rydberg for some of the test suite. I finished up the test suite,
> and have been abuse testing it for a little while.
> 
> The branch is available from here:
> http://bzr.arbash-meinel.com/branches/bzr/http/
> 
> I would appreciate it if people would download it, and try to connect to
> other branches. Especially if they have the ability to connect to a
> non-apache web server.
> 
> Just to give people a reason to test it, here are the performance results:
> 
> time bzr branch http://bazaar-vcs.org/bzr/bzr.dev/
>      original: 22m45s total
>      new pycurl: 7m18s total
>      new urllib: 9m49s total
> 
> bzr branch -r 1500 http://bazaar-vcs.org/bzr/bzr.dev/
> time bzr pull
>      original: 19m32s total
>      new pycurl: 5m23s total
>      new urllib: 7m00s total
> 
> bzr branch -r 1800 http://bazaar-vcs.org/bzr/bzr.dev/
> time bzr pull
>      original: 4m13s total
>      new pycurl: 4m07s total *
>      new urllib: 1m45s total
> 

...

> * that is not a typo, I tested it 4 times, I'm currently looking into
> the ethereal dump to see why pycurl is slower than urllib here.

Just a small update, thanks to inspecting the ethereal dump, I found
that I was sending the wrong string to pycurl it wanted:
setopt(RANGE, '0-10,100-200') not
setopt(RANGE, 'bytes=0-100,100-200')

After fixing this the pycurl times change to:

time bzr branch http://bazaar-vcs.org/bzr/bzr.dev/
new pycurl: 6m57s total

bzr branch -r 1500 http://bazaar-vcs.org/bzr/bzr.dev/
time bzr pull
new pycurl: 5m05s total

bzr branch -r 1800 http://bazaar-vcs.org/bzr/bzr.dev/
time bzr pull
new pycurl: 1m14s total

I don't know why it effects the last one so much more than the others.
But it does see a general improvement across the board. Just update to
my latest branch to get the new results. (revno 1823)

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.2 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFEtwlnJdeBCYSNAAMRAlpwAJ97WM+4DCL5GhCMoZSK6Sw7qnB6fgCbBBn0
CmA/+195rr576ilABjy26hI=
=1BWB
-----END PGP SIGNATURE-----