urllib2 Keep-alive spike

[Alex J. Champandard]

Michael Ellerman wrote:
> I've been meaning to look at whether it'd be good to use urlgrabber in
> its entirety, but I haven't had time. Are you familiar with it? It
> looks like it might get us some of the things we want, and having it
> maintained as an external project is better in the long run I think.
>

I've never looked at urlgrabber before, and it's just simply lucky I 
spent long enough time "solving" this problem with an search engine 
whose name starts with G and finishes with "oogle."

Anyway, the project seems completely unmaintained since 2004.  To use it 
in Python 2.4, you actually have to fix it to compile.  I added the 
details here:
http://bazaar-vcs.org/Specs/UrllibKeepAlive


The only other project I found using urlgrabber is Mercurial, and they 
have a copy checked in and presumably also self-maintained.
To be honest, this should 100% be part of urllib2 as the module is 
rather useless and terribly unreliable without it... The number of posts 
with people complaining about various connection errors with it is 
unbelievable, and from my experience, that's probably not their server's 
fault!

Anyway, enough ranting... That' about all I know :-)

Cheers,
Alex