[PATCH] performance history specification

[Jan Balster]

Hi Rob,

thanks for your comments.

Robert Collins wrote:
> On Fri, 2006-07-14 at 10:04 +0200, Jan Balster wrote:
>> Robert Collins wrote:
>>> On Thu, 2006-07-13 at 23:21 -0500, John Arbash Meinel wrote:
>> ...
>>> I'd like it in asap so that we can put a cron job on bazaar-vcs.org
>>> running HEAD with this, so we have object performance stats for all the
>>> benchmarks.
>>>
>> Hi Rob,
>>
>> I created a preliminary web page (and a little cron job on codespeak.net)
>> for showing performance numbers from the bzr.dev main branch.
>> The page is available at http://codespeak.net/bzr/perf_history/summary_bzr.dev.html
>>
>> It's designed to give an overview of the development on the performance front.
>> Comments/thoughts? (let's talk about fonts and colors later  ;)  )
> 
> I think there are distinct graphs that are useful:
> 1) overall performance, release to release. One way this could be done
> is by averaging the metrics for each sample to get a single figure, then
> graphing them all, and marking releases with a vertical mark of some
> sort - a bold plot point, or a vertical bar, or whatever. This is
> clearly not completely ideal as the different benchmarks are on
> different scales - but I figure after a certain point we'll become
> 'done' as far as introducing benchmarks.

I'll try to come up with an overall performance chart until Tuesday. It's a pity
that we don't have benchmarks for older versions.

> 2) Detailed performance of development within this release cycle,
> probably the last week is in fact the most interesting thing to be
> looking at.

Yip, I agree.

> A few other random thoughts:
> AddBenchmark.test_one_add_kernel_like_tree - this would be better to
> show than the full python path to the test - just grab the last two
> segments after splitting on '.'.

Done

> Perhaps put the name at the left?

Done. A smaller font would be better, or maybe shorter test names :)

> Perhaps sort with the benchmarks that have the most improvement at the
> bottom - so the ones with the least improvement, or (shock) benchmarks
> that have gotten worse at the top.

Yes, I already thought about that. It's on the TODO list.

If we want to compare the benchmark figures of different branches we need some
sort of stable order, otherwise the comparison would be cumbersome. But that's
not a current topic, I guess.

> Its this sort that having only the last weeks data shown would be most
> useful for, because it gives a locality of reference to the analysis.
> 
> Possibly highlight rows that have got worse with a different
> shade/inverse colours or something.

Yes, a bit like stock market charts :)

best,
   Jan