Rev 3105: Fix #175886 by reading remaining bytes by chunks. in file:///v/home/vila/src/bzr/bugs/175886/

[Vincent Ladeuil]

At file:///v/home/vila/src/bzr/bugs/175886/

------------------------------------------------------------
revno: 3105
revision-id:v.ladeuil+lp at free.fr-20071212154335-ac7fi089uufzbkfz
parent: pqm at pqm.ubuntu.com-20071211175118-s94sizduj201hrs5
committer: Vincent Ladeuil <v.ladeuil+lp at free.fr>
branch nick: 175886
timestamp: Wed 2007-12-12 16:43:35 +0100
message:
  Fix #175886 by reading remaining bytes by chunks.
  
  * bzrlib/transport/http/_urllib2_wrappers.py:
  (Response.finish): Don't read all remaining bytes in one read or
  some buffer may explode.
modified:
  NEWS                           NEWS-20050323055033-4e00b5db738777ff
  bzrlib/transport/http/_urllib2_wrappers.py _urllib2_wrappers.py-20060913231729-ha9ugi48ktx481ao-1
-------------- next part --------------
=== modified file 'NEWS'
--- a/NEWS	2007-12-11 05:13:36 +0000
+++ b/NEWS	2007-12-12 15:43:35 +0000
@@ -25,6 +25,11 @@
      removed use characters not supported in the terminal encoding.
      (Aaron Bentley)
 
+   * When dumb http servers returns whole files instead of the requested
+     ranges, read the remaining bytes by chunks to avoid overflowing network
+     buffers.
+     (Vincent Ladeuil, #175886)
+
   INTERNALS:
 
   API BREAKS:

=== modified file 'bzrlib/transport/http/_urllib2_wrappers.py'
--- a/bzrlib/transport/http/_urllib2_wrappers.py	2007-12-06 22:46:16 +0000
+++ b/bzrlib/transport/http/_urllib2_wrappers.py	2007-12-12 15:43:35 +0000
@@ -134,7 +134,23 @@
         """
         if not self.isclosed():
             # Make sure nothing was left to be read on the socket
-            data = self.read(self.length)
+            pending = 0
+            # We can't read all the remaining data in a single read since there
+            # may be a lot to read (several MB in the worst cases). We read and
+            # discard by chunks instead. The underlying file is either a socket
+            # or a StringIO, so reading 8k chunks should be fine.
+            bufsiz = 8192
+            while self.length and self.length > bufsiz:
+                data = self.read(bufsiz)
+                pending += len(data)
+            if self.length:
+                data = self.read(self.length)
+                pending += len(data)
+            if pending:
+                trace.mutter(
+                    "bogus http server didn't give body length,"
+                    "%s bytes left on the socket",
+                    pending)
             self.close()