Measuring success/failure in the installation

[Nigel Babu]

On Tue, May 17, 2011 at 4:14 PM, Evan Dandrea <ev at ubuntu.com> wrote:
> While we have a set of unit tests and a continuous integration system doing
> system testing of the installer, we ultimately have no idea what the failure
> rate is in the real world.
>
> Without this information, we have no means of actually measuring the true
> quality of the installation experience.  It may look gorgeous and be
> dead-simple to use, but that's worthless if it's not leading to people using
> Ubuntu.
>
> I am therefore proposing that we actually measure this.  In Oneiric, I would
> like to add code to ubiquity that, once connected to the Internet, sends a
> GUID as generated by uuidgen.  At the end of installation it would send this
> again, and the pair of values in a database would constitute a successful
> installation.  Finally, it will send this value one last time, at first boot,
> to ensure that the system actually works.  From this point the GUID will be
> discarded and never used again.
>
> I will obviously make the code for this open source, and publish the results
> to a public-facing website.
>
> The user will be able to disable this functionality by preseeding a
> well-documented key. The documentation will include a brief visual overview of
> how to accomplish this, for those unaccustomed to preseeding the installer.
>
> This addition to the installer will keep us honest. With real data to hand, it
> will be very difficult to ignore the problem if ubiquity regresses in its
> failure rate from release to release.
>
> --

Hello,

I've talked to one person who had their hard disk wiped out during an
upgrade. But the person I know was a bit frustrated at that point (and
understandably so) to file a bug report with proper information.  At
this point, as Evan says, its not clear how common such incidents are,
and test suites may not be able to give us a clear picture of real
world.  I support this idea and would gladly help build the web
interface to this.

On a side note, we must remember the implications of this.  We would
effectively be tracking the installs causing a lot of press (possibly
negative).  We should probably either only opt-in to submit data or
have a very clear opt out option.

Cheers
Nigel