[MERGE] BTreeIndex intern()

[John Arbash Meinel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

So a while back I proposed something along these lines, and Martin's
comment was that intern() was a bit iffy, so it should only really be
used if we have concrete evidence.

Since I've come back to focus on memory profiling recently, this patch
ends up dropping memory consumption approximately in half during "bzr
branch bzr.dev".

Doing a memory dump of bzr after completing a bzr branch shows that we
ended up with 15,500 copies of the file-id "NEWS-20050323055033-...".

This is a little bit more than I expected, but I think it breaks down as:
 4256 revisions modifying NEWS
 6179 parent revisions (every node has 1 or more parents)
- -----
10435

So we still have an extra 5k that I can't directly explain. It is
possible we are doing something inefficient with the leaf nodes, or some
strange interaction with the LRUCache. (I know the new _LRUNode code is
causing refcycles, because it is a double-linked list. But nodes that
fall off should still be cleaned up appropriately.)

Anyway, running "bzr branch ../bzr.dev ../standalone -Dmemory" I see:

WorkingSize         138892 KB
PeakWorking         155220 KB

=>

WorkingSize          94992 KB
PeakWorking         112528 KB

Or a 50% reduction in peak memory usage.

- From what I can tell, I think we have a small memory leak somewhere,
because the current 'WorkingSize' doesn't match the size of objects I
can access using PyMemoryDump. I feel like it is something introduced
recently, but I haven't found a great way to actually debug it yet.

Anyway, the patch helps, so it is up for review.  Using slots for
_LeafNode is a small win. Given that _LeafNode has a single attribute,
and we may have 1000s of these objects, it mostly just made debugging
with PyMemoryDump easier, as I didn't have to indirect through the extra
dict. (For branching bzr.dev, we end up with about 1.4k leaf nodes for
source and target, so this amounts to maybe 330KiB of memory savings.)

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkndCzwACgkQJdeBCYSNAAM/KgCcDPe5Ic3JVcE9UGm4FAmgsmCy
rJwAoKUJJxtaeOe3hTCo/x97eTGVqpCG
=u5k4
-----END PGP SIGNATURE-----
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: 1.5-btree-intern.patch
Url: https://lists.ubuntu.com/archives/bazaar/attachments/20090408/e79debe2/attachment.diff