[merge] benchmark caching

[John Arbash Meinel]

Robert Collins wrote:
> On Wed, 2006-08-09 at 11:40 -0500, John Arbash Meinel wrote:
>> I originally tried to break the patches down into incremental updates.
>> But since I didn't get any responses, and I had to make another update,
>> I'm just submitting the whole thing to be reviewed.
>>
>> summary:
>>   new functions on Benchmark, which can grab a cached kernel-like tree
>> in various states. Makes test setup time *much* more reasonable. It also
>> allows the cache to be preserved between runs.
>>   I also added support for creating the kernel-like tree somewhere other
>> than '.'. Which will help for certain tests that need to create more
>> than one tree.
>>
>> new changes:
>>   The old code base had just done a 'bzr add', which yields a hot
>> hash-cache, so for tests to be consistent, we need to warm it up after
>> copying the tree.
> 
> I'm -0 on the caching, +1 on the new benchmarks.
> 
> 
> I'm -0 on the caching because I think that there is too much risk of
> generating unexpected benchmark results - ones that do not match what
> the code *appears* to do. For instance - you've had to add code to warm
> up the hash cache. 
> 
> I'd rather we made doing commits and builds faster, than stop doing them
> in each benchmark.
> 
> Rob


Well, there is one other benefit that I haven't mentioned. And that is
you can construct a cache dir with a custom kernel sized tree. (For
example, you can use the real kernel).

Without any form of caching, running any sort of post-one-commit kernel
test really is too painful.
I am working on making commit fast. But in the meantime, I really need
something that makes it so I can have fast turnaround while I try
different algorithms.

I'm not trying to be a weany, but waiting 10 minutes every time I want
to check if my algorithm is faster/slower is really crappy versus
waiting ~1 min to get the same information. (It takes approx 100s to
build a 1-commit kernel tree from scratch, about 2min. If you want to
run a couple tests that is 2min * N tests just for setup, rather than
20s*N tests for setup)

If you want, we can have the pqm run with caching disabled. I can
certainly change it to add a --no-cache flag. And I would be happy to
run the benchmarks that way from time to time.

But I think we really need a cache, if we want to make it easy to run
the benchmarks.

John
=:->

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 254 bytes
Desc: OpenPGP digital signature
Url : https://lists.ubuntu.com/archives/bazaar/attachments/20060809/72288369/attachment.pgp