Initial benchmarks for checkout performance improvements

[Matthew D. Fuller]

On Wed, May 30, 2007 at 12:58:39PM -0400 I heard the voice of
Aaron Bentley, and lo! it spake thus:
> 
> Timings are hugely variable.  One run may be 5x faster than the
> next.  So I'm running each test 10 times and reporting the best
> performance.  Raw scores are attached.

Here's a few ministat runs over the wall times of cp-vs-transform and
transform-vs-checkout:


% ./ministat -w73 cp transform
x cp
+ transform
+-------------------------------------------------------------------------+
|                                                                        +|
|               x   xx                                            ++     +|
|x      x       x  xxx      x                                     ++ + +++|
|         |______A_M____|                                          |__A__||
+-------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x  10         2.183         12.05         9.045        8.0992     2.7580298
+  10        26.227        28.912       27.8805       27.6705      1.092618
Difference at 95.0% confidence
        19.5713 +/- 1.97097
        241.645% +/- 24.3354%
        (Student's t, pooled s = 2.09768)


% ./ministat -w73 transform checkout
x transform
+ checkout
+-------------------------------------------------------------------------+
|                 +                                                       |
|                 +                                                       |
|xx               +                                                       |
|xx               +                                                       |
|xx               +                                                       |
|xx               ++                                                      |
|xx               ++                                                     +|
|A|   |___________M_____A________________|                                |
+-------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x  10        26.227        28.912       27.8805       27.6705      1.092618
+  10         91.19       307.604       92.3145      114.4346     67.898444
Difference at 95.0% confidence
        86.7641 +/- 45.1172
        313.562% +/- 163.052%
        (Student's t, pooled s = 48.0177)



So, checkout is significantly more worse than transform, than
transform is than cp (just try running THAT sentence past an English
teacher...).  That outlier slaughters checkout, though.

With the one wild outlier on checkout removed, it's down to slightly
less worse:


% ./ministat -w73 transform checkout.1
x transform
+ checkout.1
+-------------------------------------------------------------------------+
|                                                                   +     |
|                                                                   +     |
| x x                                                               +     |
|xxxx                                                              ++     |
|xxxx                                                              ++  + +|
||AM|                                                              |MA_|  |
+-------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x  10        26.227        28.912       27.8805       27.6705      1.092618
+   9         91.19        97.264        92.281     92.971333     1.9787978
Difference at 95.0% confidence
        65.3008 +/- 1.5251
        235.994% +/- 5.51163%
        (Student's t, pooled s = 1.57311)



-- 
Matthew Fuller     (MF4839)   |  fullermd at over-yonder.net
Systems/Network Administrator |  http://www.over-yonder.net/~fullermd/
           On the Internet, nobody can hear you scream.