London sprint report / performance

[Robert Collins]

I thought the sprint in London was a spectacular success: We really
documented the expected costs of various operations and questioned some
of our assumptions.

We've a lot of material from that week which needs to be captured, to
allow folk that were not there to contribute over the coming weeks and
months. I have draft text for
add/annotate/gc/revert/incremental-push-pull that I'll be putting up for
review today.

We ballparked the overall effort at about 3 man-months. This was an
entirely unscientific guesstimate, just to get the size of work we're
talking about : and the answer is *lots*.

We also got a bunch of other hacking done, including oohing and aahing
at demos of the the two GUI Summer of Code projects, design discussions
on the crypto-repository Summer of Code project and more!

So where are we at for performance, and whats the way forward?

We've completed a group analysis of approximately 20 core use cases,
with each covering several variations (partial-tree operations, network
operations etc).

From that we've got a list of core questions to answer (e.g. whats the
most efficient way to detect partial-changes to two historical trees,
and whats the most efficient way to write new files and directories to
disk). More on that in a separate email - I need to transcribe a
photograph ;). These are research topics for those with a
measurement/theory bent.

We've also got a list of things that *have* been answered, and which we
are ready to start implementing.

A very important thing was the discussion about how to incorporate these
changes into the code base. There are many steps we want to take; some
of these are compatible in terms of 'concept' with the current code and
some are not. The ones that are can probably drop in as optional
incremental improvements in new repository or tree formats. The changes
that are not really compatible with the current codebase - e.g. changing
what gets gpg signed - are likely to cause interoparability issues where
upgrading to such a change will make it hard or even impossible to share
your branch with someone that has not upgraded. E.g. that you cannot
export your data to an older disk format due to fundamental model
changes. We we proposed in the sprint is that if/when such a thing
arises we will put it in the experimental format, which is not stable.
We'll aim to produce only *one* format bump that has such
interoperability issues over the course of this performance roadmap.
This means that some operations will appear to not improve in
performance as merges to fix them come in, but their improvements will
wait in the wings. Now, we're proposing this because we felt it was the
best thing for you, our users, to be able to interoperate casually with
mixed versions while we sort out all this plumbing. If anyone disagrees
with this approach, please say so!

On a related note, I've sought, and obtained the go-ahead to spend the
next 5-6 months primarily focused on bzr performance as my role at
Canonical. (For those who don't know, I'm not part of the Bazaar team at
Canonical, so there isn't a presumption that I get any *work* time to
hack on bzr :). This go-ahead means I get such time!). My goals over
this period are to act as a co-ordination point and to do anything I can
to help folk interested in improving the performance contribute to that
effort - which can take many forms. I'm going to put together some wiki
pages to help drive community effort over the next few days. In order to
avoid conflicting requirements I don't plan to be a release manager
during this time: I think that the performance effort is a strategic
one, and while releases need to keep happening, many of the units of
work needed to achieve serious performance wins will span multiple
releases. I want to be in a position to easily argue with the release
manager about things :). I will be spending considerable time writing
code for the performance effort as well, but I'm placing this
co-ordination and streaminling role above my personal coding time,
because I believe we can only achieve a massive performance boost via
major parallelisation of our development efforts : and that means
everyone interesting in performance needs to have the resources at hand
to contribute without having to invent their own wheel.

In short - fixing performance looks doable, highly parallelisable, and
the folk at the sprint had a clear picture of what that looks like. Once
I have some of the photos transcribed, I'll try to present a single
comprehensive picture of this - as I understood it anyhow :).

Cheers,
Rob


-- 
GPG key available at: <http://www.robertcollins.net/keys.txt>.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : https://lists.ubuntu.com/archives/bazaar/attachments/20070531/bea8bd7c/attachment.pgp