errors.ubuntu.com: support for Server?

[Evan Dandrea]

Hi Robie,

Thanks for kicking off this discussion.

On Thu, May 16, 2013 at 12:10 AM, Robie Basak <robie.basak at canonical.com> wrote:
> that will cause us to misinterpret the results? Is there any need to
> categorise the reports at submission time (eg. perhaps ask the reporter
> a question)?

I'd prefer that we kept with the existing model of not asking
questions, as this has been core to the redesign of apport and the
development of the error tracker. Two points on this:

1) We have been historically quite bad at understanding our audience
when programmatically communicating with them.

"Thanks for reporting this bug. It seems you have unity running. Is
the issue you are reporting is related to unity itself rather than
compiz?"

What's Unity? What's Compiz? Even if I know what both of those are,
how on Earth do I know which one is responsible for my bug? I'm just
going to close this thing because I don't know what the correct answer
is.

2) Any extra bit of work you put between a user and submitting an
error report will cause some of those users to not send the report.
They'll be hesitant to send subsequent reports if the process was
previously onerous.

> What should the UX be? How should we protect privacy and avoid reports
> inadvertently being submitted by users who do not wish to submit them?
>
> Should we have different UX defaults between the development and
> production releases?
>
> If the system is opt-in (which I assume it will be), will we have enough
> users opting in for the system to be useful?

Why do you assume an opt-in system? Error reporting on the desktop is
not opt-in, and for good reason: our data would be heavily skewed to
the types of people who already know that the error reporting
functionality exists, can find the toggle to turn it on, and are so
passionate about providing error data that they do turn it on.

On the desktop, this is handled by a checkbox that's ticked by default
with the heading:

"Send an error report to help fix this problem."

I think we should find a solution with that balance in mind. I've
suggested in the past that we do this at installation time with the
default option being to enable error reporting. So in the case of
debian-installer, a d-i component that asks whether you want to
disable error reporting. In the case of juju and maas, I'm not sure.

> Some implmentation ideas:
>
> Production server operators may want to vet reports before agreeing to
> send them, so an interactive CLI like aa-logprof(8) or apport-cli(1)
> that allows users to see exactly what they are submitting would be
> useful. So essentially a CLI equivalent of the GUI error reporting
> dialog.

The documentation for apport discusses one possible approach:

http://bazaar.launchpad.net/~apport-hackers/apport/trunk/view/head:/README#L51

The problem I have with this is that it's a very manual step. I'm not
convinced many people who would otherwise be okay with automatic
reporting will actually notice the message and follow through. It also
delays our knowing the frequency of serious issues by relying on
server administrators to log in and process the reports.

> Or perhaps some kind of remote ssh enhancement to the existing GUI error
> reporting dialog, so users get the same UI on a desktop machine that
> connects to a server machine, picks up the crash reports there generates
> and sends reports via the local GUI?

That assumes they're running Ubuntu everywhere. It's presumably
confusing as you're suddenly getting error reports out of the blue.
We've been bitten hard by this with the "system internal" error
reports that pop up, confusing users who thought they hadn't actually
done anything.

But most importantly, it's extra work for the end-users to get going.
Lets make the barrier to sending these reports as low as possible.

> An update-motd enhancement to notify server operators that crash reports
> are pending. This could provide the command to type to enter the
> interactive CLI to submit them and a reference to a manpage with more
> details on how to turn off local crash report collection.
>
> A bash prompt enhancement present only during development, which prompts
> users with a shorter version of the motd prompt. Or would this be too
> intrusive? I thought of it because I imagine users not logging in much
> to test servers, and so might not see the motd prompt (I rarely see the
> motd prompt when I test, since I use ssh shared connections and
> short-lived cloud instances).
>
> Other thoughts:
>
> Right now I feel that the information we get from users about Server
> quality is poor. As always, some bug reports are awesome, are from
> competent submitters, and highlight real problems which we need to fix.
> But many bugs filed appear to be automatic and in response to postinst
> failures due to local misconfigurations. I think this overrepresents the
> set of users who are experimenting and underrepresents the users
> who are using Server in production. I'm not sure we get any other real
> feedback about quality right now, apart from general inferences that we
> can draw from talking to people. So if we think it's worthwhile, I'd
> love to see better sources of this information.

+1 :)

In desktop-land we found a lot of very serious issues that would've
been unknown to us, because they showed up in non-developer behaviours
or they happened post-release. Prior to having
https://errors.ubuntu.com, we were missing information on the
frequency of errors, which crucially told us how important issues were
relative to one another.

One such example is that upgrading from 12.10 to 13.04 was broken
unless you first ensured you had all the 12.10 updates installed.
Nothing forces you to do this and so a lot of people found themselves
unable to upgrade at all:

https://errors.ubuntu.com/problem/27a5550c8a96660e9b19630655aa1c613772e29b

Because we had fixed the issue in a 12.10 update, it was assumed that
the job was done. The error tracker shows us that by not following the
bug fix with a discussion of how we can force installing 12.10 updates
before attempting the 13.04 upgrade, we failed our users at least
31,000 times.