[MERGE][RFC] further add performance improvements

[John Arbash Meinel]

Robert Collins wrote:
> On Tue, 2006-05-23 at 23:16 -0500, John A Meinel wrote:
>> John A Meinel wrote:
>> ...This change actually dropped the time by 5 minutes. We lose a little bit
>> of compression, but not much. And our read time is within a factor of 3
>> of a C XML implementation. (Our write time is about the same).
>>

...

> 
> The second thing here, is that we should be clear when we are
> experimenting to learn the space, and when we are testing for code that
> we want included in the next format bump - the sorts of things I'd look
> for as a reviewer are quite different between these intents :)
> 
> 
> Rob

Well, I just tried something else, and I found that I can shave off
another 25% of the 'bzr add' time for the kernel-like tree. Which on my
machine, brings it down to 4.2s.

I noticed that a lot of time is spent in "ElementTree._write". Partly
because of the

If you grab my 'rio_inventory' plugin, I added a '_xml_write' C++
extension (which uses Boost.Python).
It isn't as feature complete as ElementTree._write, since it doesn't
support namespaces, but we don't use them, so I didn't lose any sleep
over it.
It uses compiled versions of _escape_cdata and _escape_attrib, though it
calls back into python to do unicode=>utf8 conversion. I also try to use
a C++ buffer, rather than a python string buffer while creating the
serialized output, and then I dump the results to the file after each
entry is created.
It still expects everything to be a cElementTree object, rather than
working directly on the Inventory entries. But having played with it a
little, it would be pretty easy to write one that worked directly from
the Inventory object.

So my benchmark times came out like this:

test_one_add_kernel_like_tree                       OK  5937ms/13554ms
test_one_add_kernel_like_tree                       OK  4272ms/13203ms

Which is 71% the speed.
If you look at the lsprof output, 'is_ignored' is still the most
expensive call, but where I used to have:
10825  10824  2046.9680  978.0250   elementtree.ElementTree:662(_write)

I now have:
1  0  744.4100 744.4100  bzrlib.plugins.rio_inventory:28(_write_element)

Which says to me that writing out the inventory dropped in cost from
something like 2s down to 740ms.

This isn't an end-all be-all solution. Ultimately, we will want to
change how we write out inventories. If we write them out as
per-directory chunks, it won't help the 'bzr add' for a whole tree time,
but it should help out any future commands, since they would only have
to update that directory, rather than the whole tree.

The biggest thing I like about these changes, is that I was able to do
them in a plugin. Which means that someone can optionally compile
certain parts of the bzr codebase, and get better performance. But the
core code doesn't require compiling anything.

John
=:->


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 254 bytes
Desc: OpenPGP digital signature
Url : https://lists.ubuntu.com/archives/bazaar/attachments/20060527/d4366dc8/attachment.pgp