Disabling whoopsie by default in the 12.04.1 release

Steve Langasek steve.langasek at ubuntu.com
Mon Aug 20 19:32:25 UTC 2012

Hi Seb,

On Mon, Aug 20, 2012 at 12:36:07PM +0200, Sebastien Bacher wrote:
> Le 20/08/2012 12:26, Evan Dandrea a écrit :
> >Apologies. It was not my intention to imply you accepted the
> >decision, just that your team picked a number you would be happy
> >with, and we came up with data that indicated we were below that.

> Right, we picked a "3 dialogs showing up in a week" as a max rate,
> the number you guys came up with is under that but:

> - you took the number on a 90 days period I think, knowing that most
> of the common issues would be stopped after 3 reports by apport.
> It's likely that the number of the first weeks of use is much higher
> than the one you came with and it's the one make the first
> impression (if people give up after a week because Ubuntu is too
> buggy we just lost them even if the annoyance would have gone down
> over time)

Sorry, but I feel like this is a case of moving the goal posts.  We
discussed what would be an acceptable per-week average.  I would not be ok
with drawing a line saying that the average user's *maximum* number of
crashes for any week must be 3 or below, because any user who in one week
had a single repeating crash, plus any single other crash, would miss this
limit.  If a crash is affecting users repeatedly, that's important
information to know, and we shouldn't discard this data to keep the user's
dialog count artificially low.

It seems to me that there are some answers the crashdb could give here that
might be useful:

 - What is the average number of crashes reported by a machine during the
   first week that the machine is seen?  (measuring the initial experience
   vs. the post-install experience)
 - What is the percentage of systems that have a high initial rate of
   crashes, which continue submitting crash reports after a week / a month /
   three months?  How does this percentage vary with the initial crash rate?
   (approximation of the effect of crash dialogs on user retention)
 - How has the initial crashiness changed over time / what is the effective
   crash rate if all 12.04.1 updates are applied?  (measuring the
   effectiveness of the 12.04.1 work on improving the user experience)

Perhaps having these answers would give us all greater confidence in whether
whoopsie is working the way we expect?  Are there other questions that you
think we should get answers to from the crashdb?

> - we still didn't explain that 90% drop showing in a lot of the
> reports' curves

That's not true at all; I gave you the explanation on IRC as part of the
discussion.  There is a bug in apport which causes it to randomly submit
incomplete crash reports, severely impacting our ability to correctly bucket
the information being submitted.  It does *not* impact the numbers of
reports submitted: any analysis of the perceived crashiness of the desktop
is unaffected by this bug.

The bug does have the following other implications:

 - The actual user experience benefit of fixing a crash shown on
   errors.ubuntu.com is roughly 10x what it would appear to be based on the
   graphs.  This means, for instance, that with approximately 75,000
   reports/day, the top crash of the day for 12.04, with a count of 342,
   probably accounts for 4% of the crash reports from users - not .4%.
 - Crashes in packages with heavyweight apport hooks are likely to be
   underrepresented in the list of top crashes, because they're more likely
   to have incomplete data submitted.  It's impossible to quantify this
   effect without just fixing the bug.
 - Infrequent crashes may fail to be bucketed at all, if the only crash
   reports sent for the crash are incomplete ones.  However, statistically
   speaking any such crashes are likely to be ones we aren't going to work
   on anyway because they have such a small impact.

Anyway, this is an important bug to fix and an SRU of apport is being
prepared for it.  This will improve our accuracy in collating data from
crash reports.  But it does *not* affect the accuracy of our raw numbers
regarding crash submissions.

> - the 0.4 report/week/user just doesn't match real life experience,
> nautilus and some other services generate a report every second
> logout, that alone should put us over that number if you consider
> users restart their computer once a day, and we didn't start
> counting any of the "real" issues...

That hasn't been my experience.  I've *never* seen a crash report from
nautilus on logout.  So it seems this is firmly in the realm of anecdotal
evidence, because we don't know whose experience is the typical one.

> Well anyway, as said let's move on, but I still think we took a
> non-decision to keep things the way they are based on approximative
> datas

It wasn't a non-decision at all.  It was an affirmative decision to not
change apport because the data does not support making a change.

All data is imperfect and should be questioned, and I've suggested above
some ways that this particular data can be refined if you think it's
suspect.  But we did use the best available data at the time.

Steve Langasek                   Give me a lever long enough and a Free OS
Debian Developer                   to set it on, and I can move the world.
Ubuntu Developer                                    http://www.debian.org/
slangasek at ubuntu.com                                     vorlon at debian.org
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <https://lists.ubuntu.com/archives/ubuntu-release/attachments/20120820/c1f7d70f/attachment.pgp>

More information about the Ubuntu-release mailing list