Disabling whoopsie by default in the 12.04.1 release

Tue Aug 7 06:14:51 UTC 2012

Hi Seb,

On Mon, Aug 06, 2012 at 06:27:52PM +0200, Sebastien Bacher wrote:
> Let me change the angle of my suggestion, and say "we can't keep the
> LTS with that number of error prompts", that's my position.

> The possible way out listed so far:
> - turn off whoopsie
> - increase our involvement on bugs fixing
> - drop buggy softwares (not really practical in the LTS, I doubt we
> can drop update-manager or jockey like that)
> - other idea you got

> We need to pick one and make it happen, i.e we can't say "we should
> fix the issues" and not to do, so I would suggest that we either
> - allocate extra resources and focus to fix the issues on the LTS
> or
> - turn off apport

> (I don't think dropping software is practically possible)

Thanks for this reframing of the question, I think that's very helpful.

In an earlier message, Martin spoke of whoopsie being a "great experiment".
Let me say that this was never envisioned as an experiment; the intent here,
as discussed at UDS-P, was to provide a solid and low-friction error
reporting system that could be kept enabled for stable releases /over the
long term/; it was a core design goal to capture information about the
crashes affecting users running the stable releases, so that we could
properly prioritize our work around SRUs.  If that's not what's happening, I
certainly agree that we need to address this.

If there was this much disparity between the rate of error dialogs and the
rate of SRU bug fixes, it would have been nice to make this a focus for
12.04.1 work.  But that's water under the bridge now.

However, I have a hard time judging what the correct solution here is
because the information available remains rather incomplete.  I have not
heard the same feedback that you have about precise being buggy, and I'm not
sure that's actually representative of users' experience or if this is a
self-selecting sample - and if the latter, whether there are other common
factors that make the experience much worse for these users.  It would be
incredibly ironic if we turned from the path of acquiring a data-driven
understanding of Ubuntu quality solely in response to anecdotal feedback!

errors.ubuntu.com currently shows an average number of crashes per day of
1.39, which I agree is unreasonably high.  But this number doesn't reflect:

 - users who experienced no crashes
 - users who experienced crashes that they chose not to report

In fact, it doesn't even tell us how many users are included in the set
being averaged, because it only shows the top crashes.  Are there 500 more
crashes per day that get cut off the bottom of the report? 5000? 500,000?

Evan, can you provide some clarity here?  How many users is this average
taken from?  What's the standard deviation on the number of crashes per day? 
What's the turnover of users reporting crashes from day to day?  (I.e., it's
quite different if the same 5000 users are reporting 1.4 crashes per day,
than if 5000 *unique* users are reporting 1.4 crashes each day.) These are
the kinds of information I think we need to have in order to make an
informed decision.

If we can't quickly answer these questions, I think that's a strong argument
for turning whoopsie off *until we can*, with the intent of re-enabling it
once we're sure that we're on track.  Yes, that would be a setback for
gathering data about the most severe bugs; but the doubt here is about what
it's costing us (in terms of user experience) to gather that data and
whether we're making effective use of it in practice, and that's something
it's really important for us to know.  Just as long as we bear in mind that
the goal is to work out the kinks and get it re-enabled - among other
things, whoopsie is crucial to us improving our SRU process, and that
definitely requires it to be on in stable releases.

> >I don't see how uncaught Python exceptions are harmless. Even if they
> >were, codifying this belief of harmlessness in a try, except, pass
> >block is a very small patch.

> It is, yet many of those sources are unmaintained and that didn't
> happen in the 3 months between 12.04 and 12.04.1, we need to catch
> up with reality and admit that we don't keep up with issues.

I don't know what sources you're alleging are unmaintained here.  If these
are packages that are contributing substantially to the daily error rate
across all our entire userbase, then they're almost all going to be packages
in main, and those are not unmaintained.  If the level of maintenance hasn't
led to an adequate reduction in the number of python exceptions, I think
that's because this hasn't been made a priority - and that doesn't surprise
me since this wasn't brought up as a problem before now.

Without committing to any particular solution yet, I think we need to first
come up with an objective metric of what we think is an acceptable user
experience, then figure out what it takes to achieve that, and decide if we
think that's feasible.

> >Turning whoopsie off for precise leaves users without an explanation
> >when any kind of application failure occurs. It means that we cannot
> >measure stability of 12.10 against 12.04. It completely leaves us in
> >the dark on the state of 12.04 after the .1 release. It's really going
> >at the messenger with a hatchet.
> That's what we have been doing for 8 years, it's not perfect but
> it's not the end of the world, I guess we could do it for another
> release?

What would prevent this same argument from being made in another 6 months'
time about 12.10?  Furthermore, 12.04 being an LTS, it *is* supposed to
remain a target of bugfixing for some time to come; if errors.ubuntu.com is
providing important information about what our bugfixing priorities should
be, then that's true for LTS SRUs as much as it is elsewhere, and we should
carefully weigh the loss of that information.

> >The users who are talking to you about it. I suspect these are fairly
> >technical people who do not need an explanation about what just
> >happened when an application disappears - feel free to refute that
> >though. My understanding is that our target market is general
> >consumers though.
> No, they are normal non technical users who get 5 prompts on login
> while they didn't start doing anything and when they box is working
> normally and in a clean state because we collect i.e random shutdown
> issues and are showing them those after next book.

In practice, are the dialogs shown on login after shutdown the main problem? 
If so, there might be a way to mitigate this particular problem without
hobbling whoopsie entirely.

-- 
Steve Langasek                   Give me a lever long enough and a Free OS
Debian Developer                   to set it on, and I can move the world.
Ubuntu Developer                                    http://www.debian.org/
slangasek at ubuntu.com                                     vorlon at debian.org
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <https://lists.ubuntu.com/archives/ubuntu-release/attachments/20120806/7c4c1fa0/attachment.pgp>