Measuring success/failure in the installation

[Evan Dandrea]

On Tue, May 24, 2011 at 8:30 PM, Kees Cook <kees at ubuntu.com> wrote:
> I have no general objection to collecting this kind of information, so long
> as it provides actual value. For example[1], connman connects back to
> http://www.connman.net/online/status.html and reports its version every
> time it establishes a network connection. This provides direct value to the
> user because their software is able to better detect if it has actually
> found a "real" Internet connection or not, and do something about it.

We do this in ubiquity as well, whereby we hit
http://start.ubuntu.com/connectivity-check.html and ensure that it
matches a stored MD5 hash.  If it doesn't, we're in a captive portal
and notify the user as such.

I've wanted to see this done on the desktop properly for a while, but
never managed to carve out time to do this myself.

> What value does the installer connect-back actually provide the user? Why
> are raw counts of any value? It would seem that reporting a full set of
> hardware details in the connect-back would actually give you better
> before/after logs ("people with FooBar wifi are never seen again"), but it
> still doesn't provide the user with immediate direct useful improvement to
> their Ubuntu experience, so I'm not sure it's worth doing.

I do believe there's value in knowing the percentage value of failure
over time, probably tied to each version or Ubuntu release.  I see
this as a way of getting a rough idea of whether or not we're actually
building towards a more stable installation experience.

This does not provide the user with an immediately tangible benefit,
but there are plenty of examples of this sort of functionality in
other operating systems (Windows, OS X), and desktop applications
(Firefox, Chrome) where the manufacturers equally do not provide the
user with anything more than the understanding that they are helping
to build a better product, if they provide anything at all.

Firefox, for example, makes no mention anywhere that it's constantly
phoning home to update the plugin blacklist [1,2].  I suspect if users
managed to discover this was happening, they would weigh any concerns
against the loss of security and ultimately decide it was worth it.  I
don't think many people are going to want to take away a tool that
could ultimately mean the difference of their install actually working
or not (because it forces us to focus on the bugs) just so they don't
send a few packets worth of mostly-anonymous data to us.  Just as
Mozilla has seemingly decided around updating the plugin blacklist, I
don't think it warrants additional UI.

I do want to propose we collect more information that would better
educate us on the type of failures that are occurring; however, I
hesitated from doing so in the initial proposal because I wanted to
first address the issue of collecting this very small bit of data with
a simple and clear discussion, without it being muddled by
non-essential details.

For example, I believe it would make sense to record the contents of
the DMI table and xvinfo, and the CD date stamp.  However, as
mentioned, I am keen to consider collecting this kind of information
separately.

I'm behind informing the user if we start sending DMI tables back, but
I think a simple ping does not cross that threshold.  After all, we
don't throw up a warning the first time they hit security.ubuntu.com.

1: https://wiki.mozilla.org/Extension_Blocklisting:Code_Design
2: http://support.mozilla.com/en-US/kb/Firefox%20makes%20unrequested%20connections