Rev 179: Remove the collapse_instance_dicts safety check. in http://bazaar.launchpad.net/~meliae-dev/meliae/trunk

[John Arbash Meinel]

At http://bazaar.launchpad.net/~meliae-dev/meliae/trunk

------------------------------------------------------------
revno: 179
revision-id: john at arbash-meinel.com-20100805192918-o021ullgt8x7esyd
parent: john at arbash-meinel.com-20100802182732-mx3jnhax514y0zrq
committer: John Arbash Meinel <john at arbash-meinel.com>
branch nick: trunk
timestamp: Thu 2010-08-05 14:29:18 -0500
message:
  Remove the collapse_instance_dicts safety check.
  
  This means that we don't have to compute parents 2x.
  
  I also determined (separately) why some dumps use so much memory for compute_parents,
  it is because some objects can be referred to a huge number of times (check out None,
  for example).
  we should consider capping the parent lists (by default, but probably configurable).
  It is pretty much impossible to actually make use of more than 100, and you have
  to Proxy them to go via o.p anyway.
-------------- next part --------------
=== modified file 'meliae/loader.py'
--- a/meliae/loader.py	2010-07-31 04:23:53 +0000
+++ b/meliae/loader.py	2010-08-05 19:29:18 +0000
@@ -366,7 +366,7 @@
         total = len(self.objs)
         tlast = timer()-20
         to_be_removed = set()
-        for item_idx, (address, obj) in enumerate(self.objs.iteritems()):
+        for item_idx, obj in enumerate(self.objs.itervalues()):
             if obj.type_str in ('str', 'dict', 'tuple', 'list', 'type',
                                 'function', 'wrapper_descriptor',
                                 'code', 'classobj', 'int',
@@ -400,9 +400,6 @@
                 else:
                     continue
                 extra_refs = [type_obj.address]
-            if (dict_obj.num_parents != 1
-                or dict_obj.parents[0] != address):
-                continue
             collapsed += 1
             # We found an instance \o/
             new_refs = list(dict_obj.children)
@@ -509,7 +506,6 @@
             cleanup()
     if collapse:
         tstart = time.time()
-        manager.compute_parents()
         manager.collapse_instance_dicts()
         if show_prog:
             tend = time.time()
@@ -570,6 +566,7 @@
             sys.stderr.write(
                 'loading... line %d, %d objs, %5.1f / %5.1f MiB read in %.1fs\r'
                 % (line_num, len(objs), mb_read, input_mb, tdelta))
+    del temp_cache
     if show_prog:
         mb_read = bytes_read / 1024. / 1024
         tdelta = timer() - tstart