benchmark selftests

[Robert Collins]

Further to the benchmark selftests...

Martin and I have been talking about how best to provide a good
moving-picture view of the benchmark results.

There are [at least] three places that this is useful. One is to catch
regressions in speed at pqm commit time. Another is to provide a
tinderbox style view for contributors to show the change over time of a
well-known machine. Thirdly its useful for each developer to be able to
see what their branch's impact on performance is.

What we came up with is to:
 - Teach the benchmark suite how to save the time the tests took
individually to a datafile, which by default will be in '.', and will be
in a human editable format. If the file exists, then the *lowest* time
for each test ever observed is recorded, not the most recent run. So it
ratchets down over time.
 - Add an option to selftest to record in a different file.
 - Add an option to selftest to cause a test *failure* if an individual
benchmark test changes its benchmarked duration by more than 50%.
 - Have PQM run the benchmark test suite with both of those options,
saving the test run in a place that is available across different
merges, which means that a commit which slows down any benchmarked
routine by more than 50% will be rejected. This large window should be
enough to avoid false positives due to other system load. the PQM
machine is dedicated to PQM so there is little system noise.

This catches performance regressions. Next:
 - Have the selftest --benchmark output the difference in time between
the recorded time and the new run.
 - Setup a cron job to run this on bazaar-vcs.org regularly and output
the result to a web page, so you can see the current times in a known
place. We'll probably want to tweak the recording for this to record
many samples so we can get graphs overe time.

Lastly, you will see with the above work the local changes in any branch
if you make something slower. Being in '.' by default it allows each
branch to be measured separately without needing special configuration
or anything.

Rob




-- 
GPG key available at: <http://www.robertcollins.net/keys.txt>.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 191 bytes
Desc: This is a digitally signed message part
Url : https://lists.ubuntu.com/archives/bazaar/attachments/20060522/fc70dac8/attachment.pgp