UDS-Q Topic: Server Bug Triage Process

[Robie Basak]

There were two issues I wanted to bring up. I apologise for not managing
to send this email before UDS - I was away last week. So this is a bit
of a brain dump.

1. Triage list does not reduce after triaging is complete.

Bugs that can be worked on out of a triage list will inevitably be
progressed. Bugs that cannot be worked on out of a triage list will stay
there. So any list which permits bugs that cannot be worked on will tend
to fill up with bugs that cannot be worked on. This can make such a list
useless. So bugs that cannot be progressed must somehow be prevented
from appearing in any triage list.

So with this requirement in mind, I think that the current issues are:

a) Many bugs currently stay on the triage list because we can't progress
them. In particular, this includes bugs that are reported adequately but
not confirmed and with no instructions to reproduce. This is a
reasonable bug state, but unfortunately triagers can do nothing with
them. I think that these bugs are what cause most of the triaging
difficulty at the moment. For the purposes of this discussion I'm going
to call these bugs "awkward" bugs.

b) Bug importance is very difficult to select for awkward bugs. This is
because the real importance depends on the impact of the bug, and at
this stage this is often unknown. Importance can of course be changed
later, but setting a bug to low importance currently makes it unlikely
that it will be looked at again soon. If a bug is reported once,
initially appears not to be severe, but consequently turns out to be
very important, it could be missed.

c) If we set the Importance to remove awkward bugs from our first triage
list, all we are really doing is shifting the problem to a future triage
list. Really the awkward bugs should not appear on a triage list at all
until they become non-awkward (eg. get confirmed, or somebody posts
further information or steps to reproduce). 

What I'm looking for is a way to say "this bug looks fine but cannot be
worked on until something happens that is out of the control of a
triager". Currently we keep state only in bugs, which is perhaps a good
idea. But it seems that we're unwilling to mark this state as such in
the bug, perhaps in order not to discourage the community from
submitting them. What would be really nice would be some kind of tag or
marker that automatically disappears as soon as a bug is changed in any
way so that it gets the attention of a triager again.

So our options are: 1) maintain state outside of the bugs themselves, 2)
mark the "cannot be worked on state" inside the bug using a tag, or 3)
be forever plagued with bugs in our triage lists


2. Automatic submission of package maintainer script failures

2a. Inadequate bug description often makes these very difficult to
triage, and they stay on the list. Often the description is so vague
that it's difficult to even prompt for further information without
turning the bug into a full one-on-one support channel which we don't
really have the bandwidth to do. I propose that these bugs are managed
separately from regular triage, and/or have a standard "inadequate
report" type standard response, but written specifically for maintainer
script failures (since the usual one doesn't seem very appropriate for
these cases).

2b. The reporters of most of these bugs with inadequate descriptions
appear not to be interested in taking them further.

2c. Most are support requests and not bugs, reducing the time that
triagers get to triage actual bugs.

2d. Desktop package maintainer scripts are expected rarely to fail, and
if they do it probably is a bug. Server package maintainer scripts often
try to set up daemons, but users customise daemons beyond what
maintainer scripts can be expected to understand, causing them to fail
and encourage reporting as bugs. And desktop users often install server
packages which then confuses the matter when they try and upgrade.
Linked with 2c, this causes a big problem for bug triage.

Of course, there is no real distinction between server and desktop
packages expect in the nature of what they are expected to do, and thus
what their maintainer scripts try to do.

I think it would be appropriate to see these bugs dealt with
automatically (boilerplate response) unless or until:

  1) It's a "proper" bug report where the submitter has actually spent
     some effort in reporting it (ie. with a bug description on par with
     what we'd expect for a manually submitted bug).

  2) Aggregrate analysis suggests that the bug is real and thus we
     should look into it.

  3) We've caught up with all the other bugs.

At this point, I suggest always changing the subject of the bug from the
automatic subject line to one that describes the real problem. This
would really help to make the real bugs stand out from the automatic
ones.

The boilerplate would of course explain the situation with instructions
on how to get the bug looked at if the reporter wants to take it further
(eg. a variant of
https://wiki.ubuntu.com/Bugs/Responses#Not_described_well)

Robie