UDS-Q Topic: Server Bug Triage Process

[Serge Hallyn]

Thanks, Robie, all well thought out and stated.  I agree with each point
here.

-serge

Quoting Robie Basak (robie.basak at canonical.com):
> There were two issues I wanted to bring up. I apologise for not managing
> to send this email before UDS - I was away last week. So this is a bit
> of a brain dump.
> 
> 1. Triage list does not reduce after triaging is complete.
> 
> Bugs that can be worked on out of a triage list will inevitably be
> progressed. Bugs that cannot be worked on out of a triage list will stay
> there. So any list which permits bugs that cannot be worked on will tend
> to fill up with bugs that cannot be worked on. This can make such a list
> useless. So bugs that cannot be progressed must somehow be prevented
> from appearing in any triage list.
> 
> So with this requirement in mind, I think that the current issues are:
> 
> a) Many bugs currently stay on the triage list because we can't progress
> them. In particular, this includes bugs that are reported adequately but
> not confirmed and with no instructions to reproduce. This is a
> reasonable bug state, but unfortunately triagers can do nothing with
> them. I think that these bugs are what cause most of the triaging
> difficulty at the moment. For the purposes of this discussion I'm going
> to call these bugs "awkward" bugs.
> 
> b) Bug importance is very difficult to select for awkward bugs. This is
> because the real importance depends on the impact of the bug, and at
> this stage this is often unknown. Importance can of course be changed
> later, but setting a bug to low importance currently makes it unlikely
> that it will be looked at again soon. If a bug is reported once,
> initially appears not to be severe, but consequently turns out to be
> very important, it could be missed.
> 
> c) If we set the Importance to remove awkward bugs from our first triage
> list, all we are really doing is shifting the problem to a future triage
> list. Really the awkward bugs should not appear on a triage list at all
> until they become non-awkward (eg. get confirmed, or somebody posts
> further information or steps to reproduce). 
> 
> What I'm looking for is a way to say "this bug looks fine but cannot be
> worked on until something happens that is out of the control of a
> triager". Currently we keep state only in bugs, which is perhaps a good
> idea. But it seems that we're unwilling to mark this state as such in
> the bug, perhaps in order not to discourage the community from
> submitting them. What would be really nice would be some kind of tag or
> marker that automatically disappears as soon as a bug is changed in any
> way so that it gets the attention of a triager again.
> 
> So our options are: 1) maintain state outside of the bugs themselves, 2)
> mark the "cannot be worked on state" inside the bug using a tag, or 3)
> be forever plagued with bugs in our triage lists
> 
> 
> 2. Automatic submission of package maintainer script failures
> 
> 2a. Inadequate bug description often makes these very difficult to
> triage, and they stay on the list. Often the description is so vague
> that it's difficult to even prompt for further information without
> turning the bug into a full one-on-one support channel which we don't
> really have the bandwidth to do. I propose that these bugs are managed
> separately from regular triage, and/or have a standard "inadequate
> report" type standard response, but written specifically for maintainer
> script failures (since the usual one doesn't seem very appropriate for
> these cases).
> 
> 2b. The reporters of most of these bugs with inadequate descriptions
> appear not to be interested in taking them further.
> 
> 2c. Most are support requests and not bugs, reducing the time that
> triagers get to triage actual bugs.
> 
> 2d. Desktop package maintainer scripts are expected rarely to fail, and
> if they do it probably is a bug. Server package maintainer scripts often
> try to set up daemons, but users customise daemons beyond what
> maintainer scripts can be expected to understand, causing them to fail
> and encourage reporting as bugs. And desktop users often install server
> packages which then confuses the matter when they try and upgrade.
> Linked with 2c, this causes a big problem for bug triage.
> 
> Of course, there is no real distinction between server and desktop
> packages expect in the nature of what they are expected to do, and thus
> what their maintainer scripts try to do.
> 
> I think it would be appropriate to see these bugs dealt with
> automatically (boilerplate response) unless or until:
> 
>   1) It's a "proper" bug report where the submitter has actually spent
>      some effort in reporting it (ie. with a bug description on par with
>      what we'd expect for a manually submitted bug).
> 
>   2) Aggregrate analysis suggests that the bug is real and thus we
>      should look into it.
> 
>   3) We've caught up with all the other bugs.
> 
> At this point, I suggest always changing the subject of the bug from the
> automatic subject line to one that describes the real problem. This
> would really help to make the real bugs stand out from the automatic
> ones.
> 
> The boilerplate would of course explain the situation with instructions
> on how to get the bug looked at if the reporter wants to take it further
> (eg. a variant of
> https://wiki.ubuntu.com/Bugs/Responses#Not_described_well)
> 
> Robie
> 
> -- 
> ubuntu-server mailing list
> ubuntu-server at lists.ubuntu.com
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-server
> More info: https://wiki.ubuntu.com/ServerTeam