Disabling whoopsie by default in the 12.04.1 release

[Sebastien Bacher]

Le 06/08/2012 12:04, Evan Dandrea a écrit :
> Hello everyone. Apologies for the late reply; I was on holiday all
> last week. Please CC myself and Matthew on replies to this thread as
> we are not subscribed to the list.
>
> Thanks for taking the time to raise this, Seb.

Hey Evan, thanks for the reply!

> I think we could argue about whether it's showing up "too often." It's
> showing up precisely as often as the user is experiencing crashes. At
> present, this is 1.47 times a day on average (a value we wont know if
> we turn off error reporting). If consensus is that's too often, then
> we should be focused on bringing that number down by fixing the most
> pressing issues.

That's 10 times a week, that seems very often to me yes. I would be all 
for fixing the issues but in reality the resources allocated to that 
don't allow to make strong enough progress (out of software-center that 
you pointer before which is lucky to have a full team dedicated to it)

>
> https://wiki.ubuntu.com/ErrorTracker#When_there_are_multiple_simultaneous_errors
>
> If getting this into Ubuntu is a top priority for you, let me know and
> I'll raise it on my list.
Is there any chance that would land into the LTS?


> I disagree; these very well may impact users. There is simply no way
> of knowing whether a background service crashing affects the task the
> user is currently undertaking.
>
> If a DBus service crashes and is automatically restarted, what happens
> to the application that was partially through a call to that service?
> What does the graphical interface do?
>
> If we disable reporting for these issues, then we'll never know the
> frequency at which they're occurring or what's causing them. We'll be
> right back at developers having to chose between failing hard to
> surface an issue or partially recovering from it and never knowing its
> true extent or the damage it inflicts.
Well, by reviewing the errors since precise a good part of the issues 
have been judged harmless for users (often untrapped python exceptions, 
issues happening at session logout or random glitches from services like 
oneconf which do actions on regular basis and where missing a run has no 
real effect). Consensus in the desktop team is that while those are bugs 
they are often worth not bothering users about and create a sentiment of 
instability where not needed.
>
>> - errors.ubuntu.com is not good enough yet that we can properly tackle those
>> issues
> I would ask that you elaborate on this point.
(I did a bit down on the original email)

>
>> - we don't have the resources to get where we want to be in a short
>> timeframe, we are working on it but meanwhile we are impacting users for
>> things we don't make real use for
> This is patently false. There are things we want the website to do in
> the future, sure. That does not mean we should stop collecting data.

What I meant there is not that the datas are not useful, is that we are 
collecting datas about versions we will not work on (because of the 
resources allocated to the stable release mostly).
Also the suggestion is to turn whoopsie off only for one extra release 
(precise) and to keep it on on other series.

>
> I challenge the notion that the website is not usable yet. Developers
> are using it and there's already a wealth of valuable information on
> it that flat out did not exist before. Where are you going to get a
> more accurate prioritized list of errors? Where are you going to find
> a version by version breakdown of where that error manifests? Where
> are you going to find out how often Ubuntu is crashing for end-users
> (not just developers) relative to the previous version? Where are you
> going to find the problems that do not appear in Launchpad bugs?
Right, I didn't say the website is totally unusable, it has proven quite 
valuable in fact.

Again the issue are:
- we don't have enough people dedicated to work on the precise issues, 
we collect datas but don't make use of them by allocating resources to 
fix the issues
- it's getting hard to use the current errors.ubuntu.com summary because 
you can't filter out things which got a fixed rolled out from those 
which don't, half of "the main page" are probably issues we tackled but 
where the fix didn't reach users, as somebody planned work I would like 
those out of the overview because there is not a lot my team can do 
there, but it doesn't prevent us to see what are the real remaining issues.
- the user perception in a lts

>
> If I was still working on the installer, I could go to the website
> right now, punch ubiquity into the box and instantly have a list of
> what I need to focus my attention on. I *wish* I had this five years
> ago when Launchpad already had thousands of open bugs for ubiquity.
Right, that's precisely the issue ... we collect those datas but for 
who? Out of software-center who has dedicated people I would say there 
has been very little progress on the most commonly reported bugs since 
precise.

Do we have numbers of:
- the number of issues reported by day and by user since precise?
- the evolution of this number?
- the sources which have the most issues and how they progressed?

Did we "fix" apport for reporting issues every time they happen or do we 
still stop after 3 instances to not "spam" the users? If we do stop it 
probably helps to lower the number in an artificial way and makes it 
hard to see what issues are still happening or not

>
> Talk to David Pitkin and the software center team. It is my
> understanding that they use this website extensively to prioritize
> work and fix major issues. I have constantly seen peaks and troughs as
> programming error hits one of our most-used pieces of software and
> they find and fix it.
Yeah, I admit that's the one team which is having resources to make use 
of the system at the moment, they seem to have lowered their LTS 
activity as well recently though, I would guess they tackled the most 
important issue and they need to move to quantal work as well.

> I do not believe we should shy away from the problems we face or mask
> perception. If Ubuntu is more crashy than we would like, we should
> focus on identifying how we can fix that. Even if we do not have the
> resources to fix every bug, it's critical that we know what the
> biggest ones are so that we can assign work accordingly. Without the
> error tracker, we are largely sticking our finger in the wind and
> guessing.
In an ideal world sure we would fix the issues. I've been watching 
closely precise, as a 12.04.1 team member, and we made progress but 
limited one, and we had a special team focussed on the .1, with those 
people going back to "normal" work I doubt we will see much activity on 
precise issues in the next months.
>
> I would also ask that we leave Chinese whispers out of this. If
> someone has a problem with the way the system currently operates,
> please ask them to speak up here so it's recorded and we can discuss
> it.
I don't think there is much "Chinese whispers", people who have issues 
raised them, that's what I'm doing atm on this list, pitti replied to 
the thread email, Didier discussed it with you on IRC as well.

> Again, please lets not play a game of Chinese whispers. The only
> person I know who has had issues with single sign-on is Didier, and my
> understanding was that it would ask him to sign in continuously, not
> that he could not log in at all.
>
> Either way, that's an issue with Ubuntu SSO and should be tackled
> independently from this discussion.
Other people mentioned issues as well, but yes, let's focus the 
discussion here on the 12.04.1 question and keep the SSO discussion for 
another place,day.
>
>> What do people think about turning whoopsie off for 12.04.1?
> I think it's the wrong way to address the issues you've raised and
> will set us back massively, given that the bulk of our users are on
> LTS releases.
>
> As a concrete example, we've discussed using the average number of
> errors graph as a means to measure how ready we are to release the
> next Ubuntu version. If the line for the previous version is below the
> version we're currently working on, then we're not ready to release.
> That is, users of Ubuntu 12.04 are currently showing fewer errors per
> day than users of Ubuntu 12.10. Releasing the latter would be taking a
> step backwards in quality. Prior to errors.ubuntu.com we had no way of
> knowing this. Disabling the error tracker for 12.04.1 would mean that
> we wont know this and innumerable other critical data points.
Keep in mind that the suggestion is only for 12.04.1, I don't suggest we 
turn it on again in 12.10 or later, so the metrics for 12.10 can still 
be used and you can still compare to the numbers we had for precise at 
release time.

> Lets focus on making rock-solid software and not shy away from data
> that tells us there's more work do to, be it from a website or a
> handful of users.
>
Well, the issue is not to "shy away", but:
- perception matters (a lot), windows was "famous" for its blue screen 
for years (and some still joke about it) ... do we want our best version 
so far to become famous for being the OS which prompt you about system 
errors every single day?
- our users tell us that they think precise to be a lot less stable than 
previous Ubuntu version, and mostly due to the number of errors dialog 
they get (where in practice we know it's pretty stable compared to 
previous versions) ... that has a cost and is an issue we shouldn't 
dismiss the reputation we are building
- users see the prompt dialog as a major annoyance, I'm not sure what we 
can do better there though...
- we make use of the datas but in a very limited way (the only team I 
saw invest significant efforts in it is the software-center team)
- we are reducing further those limited efforts with the LTS dedicated 
team being dismissed after .1, people will also need to focus on quantal

Those cumulated makes me think we are shooting ourself in the foot, 
making users pay an high price for a limited benefit.

I would be happy to reconsider my position and proved wrong if I saw 
significant work happening on the list, including the frequent issues on 
those sources: ubiquity, update-manager, jockey, sessioninstaller, 
aptdaemon, update-notifier, software-properties, ... but so far those 
are "unmaintained", and we have a long list of issues we know about and 
don't tackle, I don't see the value of collecting datas further if we 
already don't make use of the ones we have


Cheers,
Sebastien Bacher