[RFC] Benchmark configuration handling

[Martin Pool]

On 29 Jul 2006, Robert Collins <robertc at robertcollins.net> wrote:
> On Fri, 2006-07-28 at 10:16 +0200, Carl Friedrich Bolz wrote:
> > Hi all!
> > 
> > What's the preferred way to handle configuration for benchmarks? The
> > problem is that the number of options for a benchmark run will likely
> > increase: the times that the SocketDelay sleeps per roundtrip and per
> > transmitted character are the first things, but other options might
> > follow.  (The benchmark config file probably also should contain
> > information how to produce a web page so that people can
> > easily publish results from certain benchmarking settings at
> > some point).
> 
> Erm, I think each benchmark test should have its settings fixed, I see
> no reason to configure or change them: benchmark identity would be
> valueless if they are altered. For the same reason we try not to change
> benchmarks after they are accepted into the mainline.
> 
> > So it would be nice to have a way to specify various benchmark
> > configurations and naming them (to be shown in the performance-history
> > page for example). What's a good way of doing this? We (Jan,
> > Holger and me) thought about adding a new configuration file somewhere
> > with various sections to correspond to various settings and a global
> > section to list the configurations.
> > 
> > A notes on the target group of this file: the benchmark
> > configuration probably does not need to be visible/advertised to
> > end users, i.e. should primarily be convenient to use for
> > developers/people caring for working with and producing benchmarks.
> > Probably, allowing to set (the possibly rather numerous) options from
> > the commandline is not important, is it?
> > 
> > Comments?
> 
> I think this is YAGNI. What prompted the desire to do it?

I agree. It seems to me the only time we'd want to adjust this is when
developers are working on the corresponding code, and want to see how it
will perform on a fast network or high-latency network or whatever.  And
in that case they're probably fine to just add a new benchmark case with
the particular parameters.  To do that you'd just want the benchmark
code to be reasonably clean so that it can be called with different
delay parameters.

-- 
Martin