Notes from QA metrics brainstorming

[Lars Wirzenius]

Attached.

-------------- next part --------------
2008-01-22: Distro sprint, QA metrics brainstorming
===================================================

We had a brainstorming session about QA metrics, mostly about bug
statics, for Ubuntu. Here are unstructured notes for those interested.
Note that these are ideas, not necessarily decisions to act.

    * total counts
    * turn-around times
          o average time to first reply
          o average time to fix
          o closed at incomplete 
    * fraction sent upstream
    * fraction fixed by upstream
    * crashers vs. usability bugs
    * universe vs. main 

- total bug counts
- how do we slice this? per package? per main/universe/...?
- rate of opening vs rate of closing bugs: difference should not be big

- we have raw numbers
- do we want to show a "health of Ubuntu" page that is less 
  directed at releasing than the weather report?
- what is a good way to display the information?

- launchpad has /+text and activity logs for bug state changes
=> good for answering "how long does it take to get bugs triaged?"
- might not be quite complete
=> ask for /+text to be changed to include everything from /+activity
- querying for all bugs might be very heavy: ask for a database dump
  of launchpad?
- possibly use the archives of the bugs-dist mailing list?

- have launchpad send a mail to people who file a bug for the first time,
  or file a bug for the first time against a new package
- preferably so that the e-mail is customized according to the package
- text of e-mail editable in the same way as the guided bug reporting text

- launchpad bug watches for upstream bug tracking systems can't be fixed
  if they're wrong

- we can get numbers on "needs to be sent upstream" and "has been sent
  upstream"
=> we can perhaps get community to help with the first class

- crashers: if 90% or more are from apport, we can assume the apport-crash
  tag is sufficient to identify these
- X and kernel crashes don't get apport crash reports
- searching for attachments of suitable types would help fixing this
- for a given package: what's the fraction of apport bugs?

- have results by status according to age of newest action or comment
  on bug: would make it easy to find, say, "fix-committed" bugs that are
  many months old
- "recently changed" in advanced query?

- task for new people to start on qa, housekeeping: going through
  new bugs with no attachments or replies since X days and asking the
  submitters whether they're still having the problem and later to close
  bugs that still have not received a reply

- make a per package or package type/team page with summary status of
  qa related things for that package
- find ten or so groups of packages such as "printing" and report those
  separately, in addition to per package

- integrate these to the developer-weather-report code, possibly show it
  in another instance, requires Django
- Henrik will get a new (virtual) box from IS for this
- also Brian's bug stats pages etc should go there

- timetables: we'll (= brian) start with per-package overview pages, and
  then expand that to per-group/per-team