brainstorming for UDS-N - Performance

[Zygmunt Krynicki]

W dniu 29.09.2010 20:43, Matthew Paul Thomas pisze:

> Measurement. Where can I go to see the equivalent of Firefox's
> <http://arewefastyet.com/>  for Ubuntu startup speed? Where's the
> equivalent graph for Ubiquity? For Unity? For Ubuntu Software Center?
> How much better or worse is yesterday's Natty nightly compared with
> Ubuntu 10.10? With Ubuntu 10.04 LTS?
>
> Regression handling. If any of those numbers gets worse by 5%, or 10%,
> or 25%, how soon do we realize? And what's our procedure for fixing it?

That's exactly what Linaro infrastructure team is building.

The system is composed of several components working together:

1) Data sources (tests, scripts, infrastructure to run it, etc) - lots 
of people working on this part. Currently this is mostly focused at the 
very low end of the typical linux stack (toolchain, kernel, some 
userspace performance, etc). The more people contribute data sources 
(tests/benchmarks) that can speak our standardized data format the 
better we get.

2) Storage and analysis system. This is the part I'm building. It's in 
early development but it started getting into shape recently. The basic 
idea is that you store the test result (or benchmark measurement) along 
with the whole context (software, packages, versions, etc, and 
hardware). Running a test such as 'application XYZ startup speed' on 
dozens of different hardware systems, on various software configurations 
(current, -1, +1 in development, etc). Big piles of data.

3) Reporting and notification system. This is what I'll start building 
soon. The idea is to be able do render various performance and quality 
level charts. See them across different hardware/software. Get notified 
when some unexpected changes happen, etc. It's not crystal clear what we 
want here (we need users of the system to get into a constructive 
feedback loop) but we know it's the ultimate goal. The value starts here.


Best regards
Zygmunt Krynicki