the need for a bench suite

[Martin Pool]

On Mon, 2006-01-16 at 12:50 +1100, Robert Collins wrote:
> On Sun, 2006-01-15 at 19:41 -0600, John Arbash Meinel wrote:

> > I agree. But should this new test be inside the main bzr branch. Because
> > theoretically you would want to generate/have generated a lot of data so
> > that you can test large trees.
> > And I don't think we really want bzr to carry around another 50MB just
> > to do benchmarking.

Yes, I agree it'd be valuable.  As it happens I had just started writing
such a thing.  A branch with it is here:

  http://people.ubuntu.com/~mbp/bzr.mbp.stig/

Typical output is like this:

hope% PYTHONPATH=~/work/bzr/bzr.mbp.stig ./stig
* removing old working directory tmp/make-3.80...
* unpacking source package make-3.80.tar.bz2...
> cd tmp && tar xjf ../make-3.80.tar.bz2
Initialize directory as a bzr branch (BenchmarkInit)            0.003s
Check tree status (BenchmarkStatus)                             0.112s
smart add in tmp/make-3.80
   295 files added
     3 files ignored
Smart add all non-ignored files (BenchmarkSmartAdd)             0.434s
Check tree status (BenchmarkStatus)                             0.053s
Commit (BenchmarkCommit)                                        2.027s
Check tree status (BenchmarkStatus)                             0.207s
Commit (BenchmarkCommit)                                        0.440s
Check tree status (BenchmarkStatus)                             0.205s
PYTHONPATH=~/work/bzr/bzr.mbp.stig ./stig  3.82s user 0.32s system 86%
cpu 4.780 total


"Stig" is a bit of a nerdy joke suggested by Scott, the author of HCT:

> The Stig's role on the show is to drive various cars around the Top
> Gear test track. The times he sets with these cars are kept on a
> scoreboard that keeps track of the fastest cars that have been tested.

Perhaps a less obscure name should be used.

The basic approach is that it will unpack source tarballs of various
sizes and time bzr operations upon them.  I would like it eventually to
know how to download the tarballs from public servers so that they don't
need to be included in the tree.  (The distcc benchmark does that.)  For
the moment you need to download the tarballs and put them in the pkg/
subdirectory.

I suppose there's no strong reason why it needs to be real source code;
we could perhaps just store a statistical description of the typical
composition of the tree (number of files, directories, lines per file,
chars per lines, etc.)

Obviously a lot more could be done as far as reproducible measurement,
checking other operations, keeping track of changes, etc etc.

I planned to merge this in after expanding it a little more.  Does it
look like a useful start?

-- 
Martin

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : https://lists.ubuntu.com/archives/bazaar/attachments/20060116/7723a241/attachment.pgp