Rev 3877: Fix bug #303538 by capping the max per-hunk read size at 100MB. in http://bzr.arbash-meinel.com/branches/bzr/1.11/303538_max_readv

[John Arbash Meinel]

At http://bzr.arbash-meinel.com/branches/bzr/1.11/303538_max_readv

------------------------------------------------------------
revno: 3877
revision-id: john at arbash-meinel.com-20081201232022-3vu5ekhfvnnzyujr
parent: pqm at pqm.ubuntu.com-20081201201721-zconkq0v7pow8nmw
committer: John Arbash Meinel <john at arbash-meinel.com>
branch nick: 303538_max_readv
timestamp: Mon 2008-12-01 17:20:22 -0600
message:
  Fix bug #303538 by capping the max per-hunk read size at 100MB.
-------------- next part --------------
=== modified file 'NEWS'
--- a/NEWS	2008-12-01 18:40:08 +0000
+++ b/NEWS	2008-12-01 23:20:22 +0000
@@ -23,6 +23,11 @@
     * Give proper error message for diff with non-existent dotted revno.
       (Marius Kruger, #301969)
 
+    * ``Transport.readv()`` defaults to not reading more than 100MB in a
+      single array. Further ``RemoteTransport.readv`` sets this to 5MB to
+      work better with how it splits its requests.
+      (John Arbash Meinel, #303538)
+
   DOCUMENTATION:
 
   API CHANGES:

=== modified file 'bzrlib/transport/__init__.py'
--- a/bzrlib/transport/__init__.py	2008-11-07 18:10:32 +0000
+++ b/bzrlib/transport/__init__.py	2008-12-01 23:20:22 +0000
@@ -797,6 +797,10 @@
         cur = _CoalescedOffset(None, None, [])
         coalesced_offsets = []
 
+        if max_size <= 0:
+            # 'unlimited', but we actually take this to mean 100MB buffer limit
+            max_size = 100*1024*1024
+
         for start, size in offsets:
             end = start + size
             if (last_end is not None

=== modified file 'bzrlib/transport/remote.py'
--- a/bzrlib/transport/remote.py	2008-11-21 00:23:54 +0000
+++ b/bzrlib/transport/remote.py	2008-12-01 23:20:22 +0000
@@ -323,7 +323,8 @@
         sorted_offsets = sorted(offsets)
         coalesced = list(self._coalesce_offsets(sorted_offsets,
                                limit=self._max_readv_combine,
-                               fudge_factor=self._bytes_to_read_before_seek))
+                               fudge_factor=self._bytes_to_read_before_seek,
+                               max_size=self._max_readv_bytes))
 
         # now that we've coallesced things, avoid making enormous requests
         requests = []