Analysis of 10 years of bugzilla.mozilla.org

[Christopher Armstrong]

On Tue, Jan 26, 2010 at 2:47 PM, Bryce Harrington <bryce at canonical.com> wrote:
> On Tue, Jan 26, 2010 at 11:36:15AM -0500, Karl Fogel wrote:
>> Matt Zimmerman <mdz at canonical.com> writes:
>> >I haven't watched the video, but scanned over the slides and found them
>> >interesting:
>> >http://commonspace.wordpress.com/2010/01/18/learning-from-10-yrs-of-bugzilla-data/
>> >
>> I agree -- that slide deck is very comprehensible.  Actually, I found it
>> easier to digest than the video presentation.  Here are the slides:
>>   http://commonspace.files.wordpress.com/2010/01/mozilla-toronto-office-2010-ms2.pdf
>>
>> Most interesting graph: correlation between the level of experience of
>> bug submitter and tendency of that bug to get fixed.
>
> I found the "Time to verify software defect" graph fascinating.  It
> shows that having more than 30 people commenting on a bug adds little to
> getting the bug verified unless the developer *really* groks the
> codebase inside and out, and can actually hinder the verification time.
> And having more than 75 commenters on a bug always makes the bug harder
> to verify.
>
> These findings correlate with my own experience.  Having a few people
> participate on a bug report can be quite helpful, but past a certain
> threshold it turns into noise.

Question every graph. Correlation is not causation. Maybe bugs getting
>75 comments is caused by something *else*, and that *other* thing is
also causing the bug to be hard to verify. For example, a race
condition bug would be hard to verify, *and* random changes to
environments tend to "fix" it, so you'd get lots of confused
commenters claiming they've found fixes and also claiming that
previous fixes don't work. So in this case, I would not say the
commenters made the bug hard to verify, but the race-condition nature
of the bug did.

-- 
Christopher Armstrong
http://radix.twistedmatrix.com/
http://planet-if.com/