Rev 3259: (jam) Reduce memory use during autopack. in file:///home/pqm/archives/thelove/bzr/%2Btrunk/

[Canonical.com Patch Queue Manager]

At file:///home/pqm/archives/thelove/bzr/%2Btrunk/

------------------------------------------------------------
revno: 3259
revision-id:pqm at pqm.ubuntu.com-20080310171308-z1cyxt7lfct91yes
parent: pqm at pqm.ubuntu.com-20080310153803-uy09bhh0ls8tavcx
parent: john at arbash-meinel.com-20080308093441-mo7ui50q8d8aq258
committer: Canonical.com Patch Queue Manager <pqm at pqm.ubuntu.com>
branch nick: +trunk
timestamp: Mon 2008-03-10 17:13:08 +0000
message:
  (jam) Reduce memory use during autopack.
modified:
  NEWS                           NEWS-20050323055033-4e00b5db738777ff
  bzrlib/repofmt/pack_repo.py    pack_repo.py-20070813041115-gjv5ma7ktfqwsjgn-1
    ------------------------------------------------------------
    revno: 3253.1.1
    revision-id:john at arbash-meinel.com-20080308093441-mo7ui50q8d8aq258
    parent: pqm at pqm.ubuntu.com-20080306120728-w54bzmmrr0ri1xfu
    committer: John Arbash Meinel <john at arbash-meinel.com>
    branch nick: autopack
    timestamp: Sat 2008-03-08 09:34:41 +0000
    message:
      Reduce memory consumption during autopack.
      We were requesting the copy_inventory process to extract the inventory
      lines for every inventory while autopacking. These were then
      stored in a list() and then thrown away. By skipping their creation,
      we also avoid processing the inventory hunks.
    modified:
      NEWS                           NEWS-20050323055033-4e00b5db738777ff
      bzrlib/repofmt/pack_repo.py    pack_repo.py-20070813041115-gjv5ma7ktfqwsjgn-1
=== modified file 'NEWS'
--- a/NEWS	2008-03-10 14:00:03 +0000
+++ b/NEWS	2008-03-10 17:13:08 +0000
@@ -87,6 +87,11 @@
 
   INTERNALS:
 
+    * Autopacking no longer holds the full set of inventory lines in
+      memory while copying. For large repositories, this can amount to
+      hundreds of MB of ram consumption.
+      (Ian Clatworthy, John Arbash Meinel)
+
     * New module ``tools/package_mf.py`` provide custom module finder for
       python packages (improves standard python library's modulefinder.py)
       used by ``setup.py`` script while building standalone bzr.exe.

=== modified file 'bzrlib/repofmt/pack_repo.py'
--- a/bzrlib/repofmt/pack_repo.py	2008-03-02 17:50:31 +0000
+++ b/bzrlib/repofmt/pack_repo.py	2008-03-08 09:34:41 +0000
@@ -642,9 +642,11 @@
         # at this point.
         self.pb.update("Copying inventory texts", 2)
         total_items, readv_group_iter = self._least_readv_node_readv(inv_nodes)
+        # Only grab the output lines if we will be processing them
+        output_lines = bool(self.revision_ids)
         inv_lines = self._copy_nodes_graph(inventory_index_map,
             self.new_pack._writer, self.new_pack.inventory_index,
-            readv_group_iter, total_items, output_lines=True)
+            readv_group_iter, total_items, output_lines=output_lines)
         if self.revision_ids:
             self._process_inventory_lines(inv_lines)
         else: