B+Tree indices: ongoing progress

[John Arbash Meinel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Robert Collins wrote:
| So, with the sketch showing promise, I think this is worth doing a
| spike/mini sprint on - if we can drop this in with the stackable format,
| it could take us a step towards addressing some of our intrinsic
| performance issues. Or not. Either way, I'm going to keep at it for at
| least a bit more.
|
| I am writing a script to test the performance of the new index, by
| comparing it to the indices from an arbitrary repository. I'm
| refactoring my adhoc stuff to be more repeatable.
|
| The first thing is that jam's new tighter compression _really_ affects
| index creation time. The following is using my copy of bzr.dev's
| indices:
|
| Baseline overhead: Read 92994 keys in 7.583
| GraphIndex: Wrote 10147567 bytes in 11.921
| BTreeIndex: Wrote 5722256 bytes in 13.481
|
| Baseline overhead: Read 92994 keys in 7.531
| GraphIndex: Wrote 10147567 bytes in 11.870
| BTreeIndex: Wrote 4104336 bytes in 21.608
|

Part of my feeling was what is this in context. Yes it is 10s slower
than the alternatives. How much of a gain do we get from it.

I certainly saw bigger compression (4.7MB => 3.3MB), and less
performance cost. So it may be 1 big pack, versus several small ones, etc.

I have ways to tweak it to be less costly, I can through out a few.

John
=:->

|
| The first set of figures are with index2 rev 7; the second with index2
| rev 8. So rev 8 is a huge increase over the current GraphIndex for
| writes, and 50% increase in time over rev 7, for a 28% reduction in size
| vs rev 7, and 60% reduction vs GraphIndex.
|
|
| Also, regarding zlib object copying:
|>>> import zlib
|>>> c = zlib.compressobj()
|>>> c.copy()
| <zlib.Compress object at 0x7f21b6ee2dd8>
|
| so my python has it - if yours doesn't, then that backs up my theory
| that it would be a hassle to use it :). Though we could try and
| fallback...
|
| -Rob

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkhq5hUACgkQJdeBCYSNAAOgqgCfbF+U3Hefdbaq/upWN+UN1WFq
XR0AnjUg2XURzJLsLbsBn92Liz+SsCRR
=gyj8
-----END PGP SIGNATURE-----