b+tree index update

[John Arbash Meinel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

John Arbash Meinel wrote:
| Robert Collins wrote:
| | I've merged john's index tweaks; this brings a dependency on a newer
| | pybloom from http://bazaar.launchpad.net/~jameinel/+junk/bzr-pybloom.
| | I've also implemented disk-spilling during writing. Basically it writes
| | out to disk every 100000 nodes, and does a merge sort with exponential
| | backoff between the in memory nodes and previously written disk nodes.
| |
| | Currently, it uses blooms for this spilled-to-disk data, but I think
| | this is counter productive and instead they should be turned off, this
| | will come later.
| |
| | The key message though, is that no matter how big your tree, the index
| | layer will cap at whatever is needed to hold 100000 node-values (about
| | 250MB I think). We could push this up or down based on some experience.
| | The most important thing about this is scaling, of course.
| |
| | I've also implemented (crudely) the missing iter_entries_prefix, so it
| | is now ready to be a full drop-in for GraphIndex.
| |
| | -Rob
| |
| |
|
| So... unfortunately some of your changes break on Windows.
|
| Specifically, you are using NamedTemporaryFile at one point during
| build, and then later on you use "transport.get(self._name)".
|
| Unfortunately, on windows, you can't actually ever re-open a
| NamedTemporaryFile even on the same process. And since a
| NamedTemporaryFile is automatically deleted when it is closed, I don't
| know the specific purpose of NamedTemporaryFile, at least on win32.
|
| This seems to be primarily with the FixedMemoryGraphIndex that you
| worked on.
|
| The quick fix is to use a regular tempfile.mktemp() and then just delete
| it ourselves when we do f.close().
|
| John
| =:->
|

And I should mention that your tests for _cancel* fail, because you are
renaming over top of an existing file. I think you need to use
"Transport.move()" rather than "Transport.rename()", but I haven't
really dug into the code yet.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkh9/L0ACgkQJdeBCYSNAAMGFwCffZ4UPMKr486BAamZTbIACVsS
eL0AoLmOJLu3DarbWn80xgrJV3E1ClNq
=RXSH
-----END PGP SIGNATURE-----