urllib2 Keep-alive spike

[Alex J. Champandard]

Hi,

As you may know, I've been having lots of problems with connection 
reliability and speed on my XP machine, connecting to a Twisted server.  
I had different issues with both pycurl and urllib transports, but I 
figured I should look into the urllib Keep-alive problem as it would be 
more useful for the project.

Basically, I've got it working in bzr using an old keepalive.py module 
from urlgrabber.  So, for future reference, here's a description of what 
I did.

1) Get keepalive.py

http://linux.duke.edu/projects/urlgrabber/contents/urlgrabber/keepalive.py
Documentation at 
http://linux.duke.edu/projects/urlgrabber/help/urlgrabber.keepalive.html


2) Edit the top of _urllib.py to create a new opener

from keepalive import HTTPHandler
keepalive_handler = HTTPHandler()
keepalive_opener = urllib2.build_opener(keepalive_handler)


3) Change _get_url_impl() to use that opener

global keepalive_opener
response = keepalive_opener.open(request)


That's pretty much it.  It works reliably and much faster...

As you can see, it's not ready for integration (I'm not sure I'm quite 
comfortable enough with bazaar for that anyway).  But what's left to do 
is to integrate the urllib2.HTTPBasicAuthHandler as it cannot work 
together with keepalive.py as it is now.


Anyway, I hope this helps.

Regards,
Alex