Initial benchmarks for checkout performance improvements

[Aaron Bentley]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Matthew D. Fuller wrote:
> On Wed, May 30, 2007 at 12:58:39PM -0400 I heard the voice of
> Aaron Bentley, and lo! it spake thus:
>> Timings are hugely variable.  One run may be 5x faster than the
>> next.  So I'm running each test 10 times and reporting the best
>> performance.  Raw scores are attached.
> 
> Here's a few ministat runs over the wall times of cp-vs-transform and
> transform-vs-checkout:

Thanks.  I didn't know about ministat, but it looks quite nice.  It's
good to have Max, Median, and Avg.

To expand a little, one of the reasons why we focus on minimum times is
concern over reproducibility.  There are so many causes of noise in a
system that it's hard to be sure that an average time is really
reflective of the performance of an operation.  However, the best time
seems more likely to be an instance in which noise made an insignificant
contribution to result.

> So, checkout is significantly more worse than transform, than
> transform is than cp (just try running THAT sentence past an English
> teacher...).

Yeah, that took a few reads to get.

> With the one wild outlier on checkout removed, it's down to slightly
> less worse:

The unfortunate thing about that outlier is that it's more likely to
reflect actual user experience.  It's the first run of a separate set of
benchchmarks, so the extra time is probably due to the cache being cold.
 I've re-run that benchmark, and the profile is similar.

On the other hand,  virtually every other run is a hot-cache run, so
it's justifiable to exclude it.

Aaron
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFGXcWR0F+nu1YWqI0RAkoKAJ9j/+P3F7bngq2eCWzgOQ35Y2aWcACdEDyb
KzLtbRAsNz0cvLmuLDYBrAM=
=FvFA
-----END PGP SIGNATURE-----
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: bench.checkout2
Url: https://lists.ubuntu.com/archives/bazaar/attachments/20070530/9274f4b7/attachment.diff