Bug workflow - a wider view

[Bryce Harrington]

On Tue, Jul 03, 2007 at 12:40:30PM +0200, Henrik Nilsen Omma wrote:
> I'd like to bring more focus on the overall process into our bug work as 
> well and the first step is to get a good overview of what we are doing 
> and what we need. I'd like to get feedback from both developers and bug 
> triagers on what procedures work for you and where the current 
> bottlenecks are. How can we make the process simpler and more uniform so 
> that more people will enjoy doing QA?
> 
> I'd prefer the discussion to not focus too much on the finer details of 
> the tools (ie. feature wishlists of Launchpad or bughelper can be 
> discussed separately), but rather the overall flow of feedback from our 
> testers and users and how that can be used most efficiently to improve 
> Ubuntu.

Hi Henrik,

One thing I think could help a lot is to give more emphasis (and
recognition) to those who are filling the tester role.  I think the
overall workflow process would go smoother if we had more testers, with
better tools, processes, and know-how.

Many bug management workflows don't distinguish between "users" and
"testers" very much.  I guess this is since in proprietary software
development, users don't really get much of a role.  One of the great
things about open source is that users *do* gain a role, and to great
effect.

However, a risk with this is that the "tester" role can be lost, or
reduced in prestige to the point no one really does it.  True testing is
hard and frustrating, but if done well it can save the developer time
and the user pain.

Users approach software testing from the perspective of "can I make this
new software I've never tried work?"  Bug reports are often a last
resort, and not something they do often.

Testers approach the problem from the different perspective "can I make
this software I know really well break?"  They are experienced in
creating bug reports and know exactly what information to include.  They
often have interacted with the developers in the past, and earned trust;
the developer knows if their tester tried the new software, found no
bugs, and gave a thumbs up, then they're good to go.  In an ideal
situation, the tester devotes themselves to testing each and every
release and update, so the developer can come to rely on them.

Good testers also take attack bugs with a stronger toolkit than ordinary
users.  They master the debugger, learn protocol capture tools, learn
about tracing tools, run package-specific tests, and experiment when
they find a bug to see if they can narrow it down beyond just "it
broke".  They may research upstream, validate workarounds, write
regression test cases, etc.

There are some similarities between testers and triagers, however the
roles are distinct.  Triagers focus on the reports, and working to
ensure they are complete, whereas testers focus on the underlying bugs
themselves.

I notice with Ubuntu there are a lot of people filling the testing role
officially and unofficially, but there are still a lot of areas where we
could use more good folk like this, or where the knowledge of how to do
good testing could be shared more deeply.  Also, I wonder if the
prestige of being a tester could be enhanced via some mechanism
analogous to what MOTU is for developers.

Bryce