Rev 41: Add some benchmark results for various flush sizes. in http://bzr.arbash-meinel.com/plugins/groupcompress

[John Arbash Meinel]

At http://bzr.arbash-meinel.com/plugins/groupcompress

------------------------------------------------------------
revno: 41
revision-id: john at arbash-meinel.com-20090304223243-xrg48jyhczvpkjxc
parent: john at arbash-meinel.com-20090304214211-rg22q09z8queeer0
committer: John Arbash Meinel <john at arbash-meinel.com>
branch nick: groupcompress
timestamp: Wed 2009-03-04 16:32:43 -0600
message:
  Add some benchmark results for various flush sizes.
-------------- next part --------------
=== modified file 'groupcompress.py'
--- a/groupcompress.py	2009-03-04 21:42:11 +0000
+++ b/groupcompress.py	2009-03-04 22:32:43 +0000
@@ -780,6 +780,16 @@
             keys_to_add.append((key, '%d %d' % (basis_end, end_point),
                 (record.parents,)))
             basis_end = end_point
+            # Interestingly, the sweet spot is 4MB, at 8 and 2 MB the total
+            # size increases... we need a better way of deciding than just
+            # random testing against a given dataset.
+            #   2MB     10.3MB
+            #   3MB      8.4MB
+            #   4MB      8.6MB
+            #   8MB     10.0MB
+            # This effects file content more than other bits, because they
+            # don't get *large* enough to overflow here. (The total compressed
+            # inventory size is only 1.5MB on my test set.)
             if basis_end > 1024 * 1024 * 4:
                 flush()
                 self._compressor = GroupCompressor(self._delta)