unnecessary QA work (was Re: dapper-live test)

[Colin Watson]

On Thu, May 04, 2006 at 10:30:06AM -0700, Matt Zimmerman wrote:
> Duplicates are a fact of life, but it is not too much to ask, no.
> Consider that it takes the bug reporter perhaps a minute to look for
> their bug, and then a few minutes more to write it up if they don't
> find it.  Then consider that all of these reports, from thousands of
> people, are funneled to a tiny development team, who must deal with
> all of the duplicates.  This means they spend less time on real bugs,
> which means less value for the users reporting them.

I used to agree with you, but after dealing with a package with a high
concentration of duplicate reports, I've changed my mind.


The assumption here is that the alternative to a duplicate bug report is
the user going away mollified, perhaps subscribing to a bug, but not
posting any comments. In my experience, this is not what happens in many
cases, particularly in cases where users simply do not have the
experience to tell reliably whether their bug is a duplicate of an
existing one or not. What happens is that the user either (a) files a
new bug which may or may not be a duplicate, or (b) comments on an
existing bug which may or may not be the same as their bug, perhaps
attaching debugging information. Each case has its failure modes; from
my sample size, I don't observe that either case is significantly skewed
towards the success mode or the failure mode.

In case (a)'s failure mode (a duplicate bug), every bug tracking system
I've ever worked with, including Malone, can deal with the situation
easily. You mark the bug as a duplicate, perhaps adding a comment, and
move on.

In case (b)'s failure mode (comment about a bug attached to a different
existing bug), I've never seen a bug tracking system that deals with the
situation gracefully. The best I've seen is the 'clone' command in
debbugs, which at least allows you to split off the bug history to
another bug and then deal with the two parts independently, but even
that isn't particularly great. I've never seen a system that allows a
developer to hide extraneous comments to avoid them causing confusion to
that developer in the future when they come to work on the bug, so the
failure mode always leaves permanent noise, unlike case (a)'s failure
mode which results in a neat list of duplicate bugs in a portlet which
can easily be ignored.

Thus, when an incorrect comment is filed, the developer is obliged to
triage that comment immediately to avoid terminal confusion; it is not
possible to leave it to a more convenient date unless one wants to use
one's inbox as a bug tracking system, because Malone does not have any
notion of what comments have not been replied to. The results of that
triage might be deciding that the comment is appropriate (in the case of
ubiquity, my gut feel is that that happens under half the time), telling
the commenter that their bug is really some other existing bug
(frequent), fixing the bug mentioned in the comment immediately
(generally inefficient because it means more context-switching from mail
handling to bug fixing and back), or asking the commenter to file a new
bug.

If you have to ask a commenter to file a new bug, it depends on whether
you think the bug is important or not. If the bug seems important, you
have to fix it immediately anyway or re-file it yourself, as commenters
don't always get round to filing a new bug on request; plus, in Malone,
you have to remember to make sure that the commenter is subscribed to
the bug, and I often forget. You can't always assume that somebody else
will file the bug if it's important; see for example
https://launchpad.net/distros/ubuntu/+source/ubiquity/+bug/42260, where
a user reported a crash affecting Vietnamese users that you can only
recover from by rebooting or by knowing an obscure command-line rune,
but which no other users would notice at all. I had to add that one to
my to-do file and make sure to remember to fix it separately, which is
less efficient than letting a bug tracking system remember about it for
me.


To summarise, duplicates are trivial for developers to deal with. The
alternative in practice ends up being people attaching comments to
unrelated bugs, which is complicated and highly inefficient for
developers to deal with. I'll take the duplicates any day of the week;
they may make the bug count rise a bit faster but each individual bug is
a heck of a lot easier to handle.

> If someone is not willing to spend a minute searching for a bug, will they
> be willing to answer questions about their problem to help get it fixed?
> Will they be willing to gather and send stack traces, debug logs, etc.?  If
> not, then I would rather receive the same report from someone willing to do
> so.

Malone's searching interface is not all it could be. For instance:

  https://launchpad.net/products/malone/+bug/33881
  https://launchpad.net/products/malone/+bug/38881

... and compare the experience of the reporter of
https://launchpad.net/distros/ubuntu/+source/ubiquity/+bug/43230.

Furthermore, it's not always at all obvious where to look for a bug.
Perhaps in the case of simple self-contained applications it's not too
difficult, but complicated subsystems like the installer or hardware
detection or font management have their bugs spread out over a number of
packages and it's not even conceptually easy for Malone to guide users
to the right place; it's increasingly infeasible to search through all
Ubuntu bugs for an issue unless you happen to be technically savvy
enough to know the right technical terms to search for (as far as I can
tell, Malone full-text searches only search the description and similar
fields, not all comments).

I don't think it's currently fair to say that users who cannot find
their bug in Malone won't be willing or able to answer questions or
gather necessary debugging information.

Cheers,

-- 
Colin Watson                                       [cjwatson at ubuntu.com]