Rev 23: Fix excessive memory consumption with fulltexts. in file:///net/bigmamac/Volumes/home/vila/.bazaar/plugins/fix277537/

[Vincent Ladeuil]

At file:///net/bigmamac/Volumes/home/vila/.bazaar/plugins/fix277537/

------------------------------------------------------------
revno: 23
revision-id: v.ladeuil+lp at free.fr-20090206144757-998yz1pc6entod17
parent: v.ladeuil+lp at free.fr-20090203202214-6bt37vi9y2pl2g2d
committer: Vincent Ladeuil <v.ladeuil+lp at free.fr>
branch nick: fix277537
timestamp: Fri 2009-02-06 15:47:57 +0100
message:
  Fix excessive memory consumption with fulltexts.
  
  * reconcile.py:
  (InventoryAncestryReconcilePacker._copy_fixed_nodes): Cache a
  single parent when using fulltet as storage kind to reduce memory
  consumption.
-------------- next part --------------
=== modified file 'reconcile.py'
--- a/reconcile.py	2009-02-03 20:22:14 +0000
+++ b/reconcile.py	2009-02-06 14:47:57 +0000
@@ -178,9 +178,6 @@
         if getattr(osutils, 'chunks_to_lines', None) is None:
             storage_kind = 'fulltext'
             convert_bytes = osutils.split_lines
-            # It seems that no matter how small we chose the following values,
-            # we just can't reduce memory consumption... near 1.4GB :-/
-            # -- vila 20090203
             min_batch_size = 16
             max_batch_size = 128
         else:
@@ -252,7 +249,11 @@
                         # better call pdb to have some clue about what went
                         # wrong.
                         import pdb; pdb.set_trace()
-                    parent_texts[key] = content
+                    if storage_kind == 'fulltext':
+                        # No need for more parents there
+                        parent_texts = {key : content}
+                    else:
+                        parent_texts[key] = content
                     parent_file_id = key[0]
         finally:
             pb.finished()