[RFC] new benchmark use cases

[John Arbash Meinel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Jan Balster wrote:
> Hi,
> 
> some of you saw the logs of the performance
> meeting on #bzr yesterday.  I am one of the
> guys contracted to help with improving bzr's performance
> and helping Martin and Robert particularly.
> 

...

> 
> Can you think of more benchmarks or suggest refinements of existing ones?


One thing I wonder. How many times is each benchmark run, and is the
time averaged/stddev calculated?

Because in my testing, running the same test 2 or three times in a row
for a 4 second test, I can get a 500+ms swing. Which has nothing to do
with any code changes.

If we can run multiple times, it might be interesting to report changes
in terms of standard deviation. (test was slower by 1.5 stddev, or test
was faster by 0.1 stddev). Since that might help remove natural
fluctuations.

Oh, and what about 'real world tests'? I know you were discussing
artificially introducing latency/bandwidth limits. Which I think are
great for reproducability. But I also think there is merit to having a
nearby machine where you checkout a fixed branch over sftp/http. (and
really a remote machine would be nice too).
There may be too many variables in a real world test. But at the same
time, it is testing what users really experience.

Otherwise the performance aspect of the new multirange stuff won't
really be discovered. (Unless we implement multi range requests for our
test HTTP server, so I guess that could be something to put in the
benchmarks).

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.2 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFEvQWgJdeBCYSNAAMRAqBcAJ9pnigg1/dV+VMG/tPFBp9bLdQAIwCfUfpj
IPNQOici00PBKc5Lfb9GuKA=
=Yc79
-----END PGP SIGNATURE-----