[merge] benchmark caching

[Robert Collins]

On Wed, 2006-08-09 at 18:02 -0500, John Arbash Meinel wrote:

> 
> Well, there is one other benefit that I haven't mentioned. And that is
> you can construct a cache dir with a custom kernel sized tree. (For
> example, you can use the real kernel).
> 
> Without any form of caching, running any sort of post-one-commit kernel
> test really is too painful.
> I am working on making commit fast. But in the meantime, I really need
> something that makes it so I can have fast turnaround while I try
> different algorithms.
> 
> I'm not trying to be a weany, but waiting 10 minutes every time I want
> to check if my algorithm is faster/slower is really crappy versus
> waiting ~1 min to get the same information. (It takes approx 100s to
> build a 1-commit kernel tree from scratch, about 2min. If you want to
> run a couple tests that is 2min * N tests just for setup, rather than
> 20s*N tests for setup)
> 
> If you want, we can have the pqm run with caching disabled. I can
> certainly change it to add a --no-cache flag. And I would be happy to
> run the benchmarks that way from time to time.
> 
> But I think we really need a cache, if we want to make it easy to run
> the benchmarks.

Well, lets pick this apart.

When benchmarking I think running all the benchmarks is not as
interesting as running the one that profiles what you want to improve.
Running all of them is interesting for version to version bake-offs, and
for pqm pre-commit checks and the like.

So I think part of the issue is a workflow thing: are you making
progressively narrower benchmarks to let you test the slow bits more
easily, or are you benchmarking from the outside every time?

All of that aside, I dont want to block or discourage you... so I would
be happier with a flag to *enable* caching, with a big warning that it
may lead to incorrect benchmarks, than with a flag to disable it.

Rob 


-- 
GPG key available at: <http://www.robertcollins.net/keys.txt>.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 191 bytes
Desc: This is a digitally signed message part
Url : https://lists.ubuntu.com/archives/bazaar/attachments/20060810/41e36665/attachment-0001.pgp