[RFC] Benchmark reproducibility

[John Arbash Meinel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Matthew D. Fuller wrote:
> On Wed, May 09, 2007 at 10:18:57AM -0500 I heard the voice of
> John Arbash Meinel, and lo! it spake thus:
>> test_read_10k_index_c          OK      184ms/    1454ms
>> test_read_10k_index_c_again    OK      150ms/    1372ms
>> test_read_10k_index_py         OK      222ms/    1518ms
>> test_read_10k_index_py_again   OK      179ms/    1406ms
>>
>> These are exactly the same test, I'm just asking it to run 2 times
>> instead of just once.
> 
> That's really odd.  What do you get with another handful of
> invocations of it?  It'd be interesting to see just how tightly they
> cluster.
> 
> Maybe lazy importing is costing you the first time through?  The
> difference between the py's is slightly bigger (43, 19.4%) than the c's
> (34, 18.5%), though 2 data points cant't really say much...
> 
> 
> Somewhat peripheral, but ministat[0] is handy for quickly slapping up an
> analysis of stats like this.  I invented some data points around the
> above for an example:

...

> [0] <http://www.freebsd.org/cgi/cvsweb.cgi/src/tools/tools/ministat/ministat.c?rev=1.10;content-type=text%2Fplain>
>     Needs libm, a quick "cc -o ministat ministat.c -lm" should do the
>     trick.  The attached patch was needed to get it built for me on a
>     Fedora 6 box; presumably something similar will do the job on
>     Ubuntu.  Yeah, I know I've beat this drum before.


This did look interesting, so I gave it a bit of a shot. But at least
Ubuntu Edgy doesn't have TAILQ_EMPTY, TAILQ_FOREACH, TAILQ_FIRST, or
TAILQ_LAST defined.
Which is funny, because they have other TAILQ functions (_HEAD, _ENTRY,
etc).

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFGQyJNJdeBCYSNAAMRApRAAKCrWFhC1msNsg8j6qTaqIiqBqQJNwCdFFGv
DCJA1i80cu+MZ9BIH6FGEHQ=
=l+Iq
-----END PGP SIGNATURE-----