Disabling whoopsie by default in the 12.04.1 release

[Evan Dandrea]

Hello everyone. Apologies for the late reply; I was on holiday all
last week. Please CC myself and Matthew on replies to this thread as
we are not subscribed to the list.

Thanks for taking the time to raise this, Seb.

On Thu, Aug 2, 2012 at 10:31 PM, Sebastien Bacher <seb128 at ubuntu.com> wrote:
> That's something quite some people raised as an issue since precise, the
> frequent whoopsie dialogs in the LTS gives users the feeling that precise is
> unstable (it seems often qualified to buggier over previous release for no
> reason out of the number of error dialogs showing up to report bugs).
>
> I've discussed the issue with different people in different teams, here is
> my try to a summary of the pro and con:
>
> Pro:
> - it gives us infos on what issues users run into
> - it gives feedback to users on what happen when a program close while they
> are using it
>
> Con:
> - it's showing up too often and giving the impression to users that the
> system is buggy

I think we could argue about whether it's showing up "too often." It's
showing up precisely as often as the user is experiencing crashes. At
present, this is 1.47 times a day on average (a value we wont know if
we turn off error reporting). If consensus is that's too often, then
we should be focused on bringing that number down by fixing the most
pressing issues.

Matthew has graciously tackled reducing the number of dialogs on the
screen at any one time, a longstanding issue with apport that predates
the error tracker. In the future, multiple errors that occur around
the same time will be grouped into a single dialog:

https://wiki.ubuntu.com/ErrorTracker#When_there_are_multiple_simultaneous_errors

If getting this into Ubuntu is a top priority for you, let me know and
I'll raise it on my list.

> - it's often showing programming errors which don't impact users (untracked
> exception from python softwares or services by example), what users get the
> most notified about is glitchs about things like ubuntuone services,
> oneconf, software-center ... most of those are not user visible issues and
> the frontend would handle the glitches fine without whoopsie notifying the
> users

I disagree; these very well may impact users. There is simply no way
of knowing whether a background service crashing affects the task the
user is currently undertaking.

If a DBus service crashes and is automatically restarted, what happens
to the application that was partially through a call to that service?
What does the graphical interface do?

If we disable reporting for these issues, then we'll never know the
frequency at which they're occurring or what's causing them. We'll be
right back at developers having to chose between failing hard to
surface an issue or partially recovering from it and never knowing its
true extent or the damage it inflicts.

> - errors.ubuntu.com is not good enough yet that we can properly tackle those
> issues

I would ask that you elaborate on this point.

> - we don't have the resources to get where we want to be in a short
> timeframe, we are working on it but meanwhile we are impacting users for
> things we don't make real use for

This is patently false. There are things we want the website to do in
the future, sure. That does not mean we should stop collecting data.

Nearly all of the changes that affect the website happen in either the
daisy codebase (the system that receives the crashes) or the errors
codebase (the website itself).

We are taking a big data approach to this problem and collecting more
information than we are using at present. Thus, the changes we make
are mostly ways to better map the flow of information to our database,
Cassandra. It is often possible to back-populate the new structure
with the existing data.

So when a new feature to the website lands, it will immediately begin
working for releases that are already very frozen, like 12.04. This
was exactly the case with the per-version breakdown on individual
problem pages.

I challenge the notion that the website is not usable yet. Developers
are using it and there's already a wealth of valuable information on
it that flat out did not exist before. Where are you going to get a
more accurate prioritized list of errors? Where are you going to find
a version by version breakdown of where that error manifests? Where
are you going to find out how often Ubuntu is crashing for end-users
(not just developers) relative to the previous version? Where are you
going to find the problems that do not appear in Launchpad bugs?

If I was still working on the installer, I could go to the website
right now, punch ubiquity into the box and instantly have a list of
what I need to focus my attention on. I *wish* I had this five years
ago when Launchpad already had thousands of open bugs for ubiquity.

Talk to David Pitkin and the software center team. It is my
understanding that they use this website extensively to prioritize
work and fix major issues. I have constantly seen peaks and troughs as
programming error hits one of our most-used pieces of software and
they find and fix it.

> I know that most of the cons are addressable but until we do address them
> the consensus form the people I talked to seems to be that the cost-benefit
> is largely not in our favour at this point so I would recommend we do
> disable it by default for 12.04.1 to minimize LTS annoyance and keep it
> enable from now on for new release (on the basis that the system will
> improve enough that we can catch up and that the perception issues is a bit
> less of an issue on a non LTS)

I do not believe we should shy away from the problems we face or mask
perception. If Ubuntu is more crashy than we would like, we should
focus on identifying how we can fix that. Even if we do not have the
resources to fix every bug, it's critical that we know what the
biggest ones are so that we can assign work accordingly. Without the
error tracker, we are largely sticking our finger in the wind and
guessing.

I would also ask that we leave Chinese whispers out of this. If
someone has a problem with the way the system currently operates,
please ask them to speak up here so it's recorded and we can discuss
it.

> Just to give some details about the errors.ubuntu.com issues mentioned
> before (I think people are going to ask what are the problems at some point
> so I can as well reply to that here):
> - it's not possible to filter out issues which have been resolved from the
> list (so it's hard to know what has been worked on and what needs work
> still)

Issues that are fixed automatically fall off the front page as users
pick up the new version. They don't fall off as fast as we would like,
but that's because http://errors.ubuntu.com also surfaces the problem
we have with getting fixes to users quickly. Fortunately, the website
gives us real numbers that the software center team can use to track
their progress in improving this.

In the iteration of the website that's in the process of being
deployed (RT 54702), on each problem page you can see a table with the
breakdown of version numbers and the number of instances of that
problem. If the version you just uploaded does not ever appear, then
you know the problem is fixed.

As mentioned privately but worth noting here, we are constantly
working to improve this. We will get to the stage where we can grey
out problems on the main page that have not occurred in the most
recent version of the software. I have a branch that has this
implemented, but it needs some polish before I'm happy landing it.

Again, this is entirely independent of crash reporting on the desktop.
We need to make 0 changes to the software on people's computers for
this to happen.

> - it's not possible to say what issue happens to what version and get stats
> of instances of the bugs by version, i.e to get datas on whether the
> situation improved or not for a bug

As mentioned above, this is landing with RT 54702.

> - some of our engineers have issues login into the system to get access to
> the infos they need to work on the bugs, that situation is still not
> resolved after some months

Again, please lets not play a game of Chinese whispers. The only
person I know who has had issues with single sign-on is Didier, and my
understanding was that it would ask him to sign in continuously, not
that he could not log in at all.

Either way, that's an issue with Ubuntu SSO and should be tackled
independently from this discussion.

> - other small issues, but I don't want to turn that email into a "list what
> is wrong currently", especially when we have people knowing about those and
> working on improving the situation

I would prefer you did, but do keep them focused on problems on the
desktop software.

http://errors.ubuntu.com is already useable to developers. Changes we
make to improve it are almost always independent of the client
software and are able to take advantage of mountains of data we've
already collected, precisely because we've kept that client software
running.

> I would add that Evan did an amazing job so far and that errors.ubuntu.com
> has been proved very useful already (we fixed at least an hundred bugs, most
> wouldn't have show up in this way using launchpad)
> (snip)

Thanks

> What do people think about turning whoopsie off for 12.04.1?

I think it's the wrong way to address the issues you've raised and
will set us back massively, given that the bulk of our users are on
LTS releases.

As a concrete example, we've discussed using the average number of
errors graph as a means to measure how ready we are to release the
next Ubuntu version. If the line for the previous version is below the
version we're currently working on, then we're not ready to release.
That is, users of Ubuntu 12.04 are currently showing fewer errors per
day than users of Ubuntu 12.10. Releasing the latter would be taking a
step backwards in quality. Prior to errors.ubuntu.com we had no way of
knowing this. Disabling the error tracker for 12.04.1 would mean that
we wont know this and innumerable other critical data points.

Lets focus on making rock-solid software and not shy away from data
that tells us there's more work do to, be it from a website or a
handful of users.

Thanks,
Evan