bazaar performance with single large project and a comparison with?git / mercurial

[Ian Clatworthy]

David Cournapeau wrote:
> Talden wrote:

>> Bazaar resolves so many of the other issues we've had with CVS (and
>> admittedly introduces a few caveats) that we're currently banking on
>> Bazaars performance improving.

I personally think that's a safe bet. For well over a year now, we have
released a new version of Bazaar once a month and every release has
improved performance in one way or another. More importantly, we
generally know what we need to do in the places where Bazaar needs
further improvement. We are continuing to make performance a very high
priority, both in terms of internal algorithm/data-structure changes and
in terms of new features (e.g. stacked branches).

> Note that there is a benchmarking tool for bzr vs other vcs (git and hg
> are supported for sure), and it is extremely easy to use (it took me 5
> minutes the first time)

As the primary author of bzr-usertest, I'm pleased you found it easy to
setup and use. It would be really nice if more people used it. It would
also help a lot if more people did their benchmarking by adding their
scripts to bzr-usertest, so we could repeat them from release to
release, from platform to platform. (Thanks to the success of xUnit, we
all seem to understand the importance of repeatable unit tests these
days, while not thinking twice about manual benchmarking.)

> http://people.ubuntu.com/~ianc/plugins/usertest/doc/usertest-tutorial.html
> 
> Unfortunately, it only tests performances wrt to "working tree size", by
> which I mean it will give you a good idea of the relative performances
> and scalability wrt number of files/size of files in your repository,
> but it does not give any indication about the scalability wrt the
> history size (which is, with network operations, the main problems of
> bzr performances I guess).

That was true in early releases of bzr-usertest but it isn't today.
Instead of passing a vanilla tar.gz file to bzr-usertest, if you pass
one with a .bzr directory inside the .tar.gz, then it *will* run the
tests - except the initial import - against the historical data, not
just against a historyless repo. (Apologies if the doc is lacking about
that feature.)

OTOH, public reporting of Bazaar benchmarking results against historical
repositories has not been done yet to the same level I reported about
historyless trees. The limiting factors there are:

1. Lack of converted repositories, at least for the same projects
   that http://bazaar-vcs.org/Benchmarks covers.

2. My priorities are currently some other important things, e.g.
   EOL support, filtered views, etc.

As a background task, I'm gradually making progress on (1), largely
thanks to another plugin I've been working on - bzr-fastimport. See
http://people.ubuntu.com/~ianc/benchmarks/repos/bzr/ for the repos
converted so far. Hmm - I guess there are arguably enough repos
converted now to get some useful data published?

I should stress that converting the repositories, running the
benchmarks, reporting the results and extending bzr-usertest isn't
particularly hard - it just takes a fair amount of time and someone
interested in doing these things. In particular, it would be great if
someone running Windows wanted to get in and help in this area. To
borrow an overused quote, "What gets measured, gets improved" and I wish
we were measuring and reporting our performance on Windows in particular
more often.

Ian C.