[rfc] bug handling priorities

[Martin Pool]

Martin Pitt gave an interesting talk last week on dealing with high
bug volumes (100,000s) in Ubuntu without getting swamped.
<http://people.ubuntu.com/~pitti/talks/efficient-bug-handling/>  I was
thinking about what this meant for bzr.

We seem to have a lot of stuff queued up in

 * patches waiting for review - I think these are tracked reasonably
well in Launchpad and Bundle Buggy, but we need to get through them
faster
 * patches needing to be finished off - there are some very old
patches that aren't merged, either because people didn't come back to
them, or they need tests, or they fail some tests
 * branches or broader items of work that aren't finished, released
and stable - these are problematic because if there's no patch
outstanding they're not visible in the previous points, and there can
be a lot of work outstanding.
 * ideas needing pre-implementation feedback - tagging list mail with
[rfc] seems to work reasonably well at getting the right attention,
but it's not always clear when there's been enough
 * https://answers.launchpad.net/bzr - I like this compared to list
posts because it makes it clear what's resolved or not
 * user questions on the list
 * bugs needing triage

In other words there's a lot of stuff that's 50% or 80% or 90% done
and we could get credit for it if we finished it off.

In pitti's view (as I remember), the right priority for working on bugs is

 1. critical bugs or regressions
 2. bugs already assigned to you
 3. bugs you can route to someone else, or otherwise get towards a
ready-to-fix state
 4. triaging new bugs

The point is that it's actually shipping improvements that helps
users, not bug paperwork.

It's not a strict queue in that you might still do a bit of triage
while you still have bugs assigned, but your time should be strongly
weighted towards the top.  Driving your assigned open bugs to zero is
more important than driving the number of untriaged bugs to zero.

So if you think about it as queues: there's little point starting to
write new code if we already have code waiting to be reviewed or
landed.  And there's probably little point triaging bugs if we're
unlikely to find new ones that are higher-priority than those we're
already working on.

Launchpad developers point out that having handoffs waiting for review
lets it slip out of the developer's consciousness, so they have an
on-call reviewer.  We don't have enough reviewers to have someone
doing it 24/5, but we could aim to do reviews before doing almost
anything else.  That'll feel bad because other stuff will be on hold,
but perhaps that just indicates there's too much stuff in flight.

Possibly making large work items visible will help us focus on
finishing them off promptly.  We could put a table in
<http://bazaar-vcs.org/Roadmap> (I started); possibly these things
should instead be bugs or specs.  That has the advantage of being more
integrated, but unless we write an API report we won't have a clear
view of what's started and outstanding.

There are a couple of sticky issues:

One is user support, where they may be easy to answer and timeliness
may help a lot if there is already an answer.  I think we do
reasonably well on the list and irc but more people should watch
<https://answers.edge.launchpad.net/bzr/+questions?field.sort=recently+updated+first&field.actions.search=Search&field.status=Open&field.search_text=>

The other is that explicitly marking bugs as "confirmed, low priority"
has a low payoff, but detecting and acting on critical bugs has a high
payoff.  Saying "thanks for your bug", removing dupes before they
proliferate, or getting more reproduction data while it's still fresh
is somewhat useful, but on the whole I suspect users would rather get
1 bug fixed than be thanked 20 times.  On the whole I think the right
thing is probably to just scan bug mail for things that look critical
and triage them, and then garden the others as a background task,
which is pretty much what's happening.

In bug gardening I typically make the summary better, fairly often
de-dupe, usually set the priority and status, and make one comment.
It could possibly be improved by a tool.

The heuristic I have in mind is

 fixing critical bugs
 >> making releases
 >> reviewing or otherwise unblocking other developers
 >> landing approved patches (possibly with tweaks or test fixes)
 >> answering user questions
 >> working on in-progress bugs or development
 >> triaging bugs
 >> starting new work

Perhaps one should think twice about starting work on a new bug or
feature if there's anything outstanding at a higher level.

-- 
Martin <http://launchpad.net/~mbp/>