[MERGE] _KnitIndex._load_data in pyrex

John Arbash Meinel john at arbash-meinel.com
Wed May 9 21:24:39 BST 2007


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

One thing that actually costs us a fair amount of time is reading the
.kndx files. The big ones are 'inventory.kndx' and 'revisions.kndx',
since they have an entry for every revision in the repository.

The attached patch writes _load_data in pyrex.

This drops the time to read a 50,000 record .kndx file from somewhere
around 900ms down to around 350ms. (not quite 3x faster).

This actually has a bit more of a direct impact for a lot of functions.

It should help things like "build_tree" which has to read a lot of
different knit files.

One example is:

  repo.get_inventory_xml(last_revision);

On a freshly locked repository, we have to read inventory.kndx, and then
build up the revision.

Without the patch, it takes 266ms, with it, it takes 104ms.

Another check is "bzr log --short -r -10..-1 > /dev/null"

This has to read at least revisions.kndx (and I think it also reads
inventory.kndx).

For a bzr.dev tree, this takes 502ms normally, and 345ms after the patch.

For a mozilla tree with 175k revisions, it changes from 6.38s to 3.78s.

The real win would be to not have to load the entire index in memory.
But I don't think that is possible in our current design.


This time I went for: bzrlib/knit_c.pyx. It would probably make more
sense to make this a private module (_knit_c.pyx).

But Robert and I still need to work out how we want to handle pyrex
extensions.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFGQi4GJdeBCYSNAAMRAvhZAKDXLsD96AIqsU3nMlSo/Lg1rqSWeQCg1FGX
Ktqnm++Z3am9oX03Nicwyo4=
=BL27
-----END PGP SIGNATURE-----
-------------- next part --------------
A non-text attachment was scrubbed...
Name: knit_index_pyrex.patch
Type: text/x-patch
Size: 112980 bytes
Desc: not available
Url : https://lists.ubuntu.com/archives/bazaar/attachments/20070509/a8af91da/attachment-0001.bin 


More information about the bazaar mailing list