bzr-usertest - a benchmarking toolkit for Bazaar and other command line (VCS) tools

Ian Clatworthy ian.clatworthy at internode.on.net
Mon Jan 28 13:24:37 GMT 2008


I'm very pleased to announce bzr-usertest, a Bazaar plugin for running
and reporting benchmarks of command line tools, and Bazaar in
particular. Bazaar already has an awesome test suite (10,000+ tests)
that can be used to verify functional correctness on various operating
systems. bzr-usertest complements this by measuring performance,
efficiency and other interesting non-functional quality attributes.

In particular, bzr-usertest has the following goals:

   1. Give users an easy way to benchmark Bazaar in their environment,
      i.e. their project on their computers/OS/network
   2. Give developers a toolset for comparing one Bazaar release to
      another in order to track improvements and avoid regressions
      over time.
   3. Give developers a toolset for comparing Bazaar against similar
      tools across a range of operating systems and project sizes.

The measurements of interest include:

    * time taken to complete user tasks
    * disk space required
    * memory used
    * network traffic sent/received.

The initial focus is speed and disk space efficiency of Bazaar. The
framework is in place though for measuring other attributes and for
measuring how alternative tools (e.g. hg, git, svn) compare. See
http://people.ubuntu.com/~ianc/plugins/usertest/doc/ for additional
information, particularly the tutorial here:
http://people.ubuntu.com/~ianc/plugins/usertest/doc/usertest-tutorial.html.

Grabbing the source is as simple as:

   bzr branch lp:bzr-usertest usertest

Please give this a go on your platform: your computer, your operating
system and your code base. I'm particularly keen to ensure this is
working on non-Linux platforms so we can begin tracking performance and
efficiency in a broader manner than we currently do. Our goal is to
ensure that every release of Bazaar is better than the one before it for
our users. Getting 'usertest' run frequently in different environments
can help us keep track of how we're going and highlight areas needing
more focus.

If you would like to contribute improvements, let me know and we can
expand the team of developers who can commit changes accordingly.

Ian C.



More information about the bazaar mailing list