[MERGE] [Bug 215426] Re: MemoryError - recv for HTTP through Proxy

[Andrew Bennetts]

Eric Holmberg wrote:
> Work-around patch for cause of MemoryError on HTTP download under Windows is attached. 

bb:tweak

Comments inline.

> === modified file 'bzrlib/transport/http/response.py'
> --- bzrlib/transport/http/response.py	2008-01-03 16:26:32 +0000
> +++ bzrlib/transport/http/response.py	2008-04-11 21:28:24 +0000
> @@ -206,7 +206,18 @@
>              limited = self._start + self._size - self._pos
>              if size >= 0:
>                  limited = min(limited, size)
> -            data = self._file.read(limited)
> +           
> +            lst = []
> +            while limited > 0:
> +                # limit reads to 4 MB for Windows problem
> +                # See bug: http://bugs.launchpad.net/bzr/+bug/215426
> +                # nBytesToRead = min(limited,1024*1024*4 + 1024*100) OK
> +                # nBytesToRead = min(limited,1024*1024*4 + 1024*128) Fails
> +                nBytesToRead = min(limited,1024*1024*4)

Our style guide says this line ought to be written as:

                num_bytes_to_read = min(limited, 1024*1024*4)

For that matter, "lst" ought to be called something clearer, like "byte_segments".

> +                lst.append(self._file.read(nBytesToRead))
> +                limited -= nBytesToRead
> +            data = ''.join(lst)

I wonder a little if a cStringIO would be a faster (and maybe clearer) way to
build up this string?

            buffer = StringIO()
            while limited > 0:
                # limit reads to 4 MB for Windows problem
                # See bug: http://bugs.launchpad.net/bzr/+bug/215426
                # num_bytes_to_read = min(limited, 1024*1024*4 + 1024*100) OK
                # num_bytes_to_read = min(limited, 1024*1024*4 + 1024*128) Fails
                num_bytes_to_read = min(limited, 1024*1024*4)
                bytes = self._file.read(num_bytes_to_read)
                buffer.write(bytes)
                limited -= num_bytes_to_read
            data = buffer.getvalue()

I don't mind either way, though.
            
-Andrew.