benchmark selftests

[John Arbash Meinel]

Robert Collins wrote:
> Further to the benchmark selftests...
> 
> Martin and I have been talking about how best to provide a good
> moving-picture view of the benchmark results.
> 
> There are [at least] three places that this is useful. One is to catch
> regressions in speed at pqm commit time. Another is to provide a
> tinderbox style view for contributors to show the change over time of a
> well-known machine. Thirdly its useful for each developer to be able to
> see what their branch's impact on performance is.
> 
> What we came up with is to:
>  - Teach the benchmark suite how to save the time the tests took
> individually to a datafile, which by default will be in '.', and will be
> in a human editable format. If the file exists, then the *lowest* time
> for each test ever observed is recorded, not the most recent run. So it
> ratchets down over time.
>  - Add an option to selftest to record in a different file.
>  - Add an option to selftest to cause a test *failure* if an individual
> benchmark test changes its benchmarked duration by more than 50%.
>  - Have PQM run the benchmark test suite with both of those options,
> saving the test run in a place that is available across different
> merges, which means that a commit which slows down any benchmarked
> routine by more than 50% will be rejected. This large window should be
> enough to avoid false positives due to other system load. the PQM
> machine is dedicated to PQM so there is little system noise.

As long as there is a way to override this. I could see a case where we
may need to do a correctness fix, which negatively effects performance.
Or we may make a test longer, which doesn't specifically change
performance, just checks for more things.

And is this 50% faster than the fastest ever, or 50% off of the last time?

> 
> This catches performance regressions. Next:
>  - Have the selftest --benchmark output the difference in time between
> the recorded time and the new run.
>  - Setup a cron job to run this on bazaar-vcs.org regularly and output
> the result to a web page, so you can see the current times in a known
> place. We'll probably want to tweak the recording for this to record
> many samples so we can get graphs overe time.
> 
> Lastly, you will see with the above work the local changes in any branch
> if you make something slower. Being in '.' by default it allows each
> branch to be measured separately without needing special configuration
> or anything.
> 
> Rob

I like the ideas, I just don't want us to be hamstrung by a commit policy.

John
=:->


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 254 bytes
Desc: OpenPGP digital signature
Url : https://lists.ubuntu.com/archives/bazaar/attachments/20060522/c557726e/attachment.pgp