Measuring success/failure in the installation

[Scott Kitterman]

On Tuesday, May 17, 2011 06:39:24 PM Steve Langasek wrote:
> Hi Scott,
> 
> On Tue, May 17, 2011 at 09:31:59AM -0400, Scott Kitterman wrote:
> > You knew you'd get this response eventually ...
> :
> :-)
> :
> > While I believe you are trying to solve an important problem, I don't
> > think this is the right way to go about it.  I do not think a design the
> > phones home by default is appropriate for Ubuntu.  I don't think that
> > opt-out via pre- seeding is acceptable even if on by default is
> > determined to be acceptable as it limits this to expert users.  Privacy
> > should be for everyone.
> 
> I believe that an installed Ubuntu system will also contact
> security.ubuntu.com by default (regardless of what mirror one selects for
> the main archive).  From the user's perspective, is there any difference in
> the information exposed via the GUID tracking method, vs. having each
> machine contacting the central security.u.c server?

There's a difference in purpose since contacting security.u.c is part of 
keeping the system functioning in a reasonable and safe manner.  The GUID 
approach is not relevant to use of the system, so I think it's fundamentally 
different.  In my opinion it's perfectly reasonable to do things that are 
needed for proper system operation by default, but communication outside the 
user's system for other reasons should be opt-in.

>  - neither method includes identifying personal information
>  - both methods can be used to get a rough count of the number of installs
>    behind a given IP address, if someone on the Canonical side chose to
>    track this (the GUID gives a direct count of new machine installs;
>    security.u.c gives an approximate count of active machines based on
>    number of apt-get update runs per day)
>  - both methods can be screened by use of a firewall / web proxy
> 
> I too am concerned that we not violate users' privacy by gathering
> identifying information from them without their consent - even if Canonical
> has no designs on using this information maliciously, I don't think even
> the temptation should be there.  But if the information being gathered
> here is equivalent to other information we already have in terms of
> sensitivity, while giving the installer team better tools to improve the
> installer experience, I would hate to see this initiative be hamstrung by
> privacy questions that don't actually benefit the user.
> 
> > I would recommend instead one more checkbox in the existing setup that
> > invites the user to provide installation success information to help
> > improve Ubuntu and it should be unchecked by default.
> 
> This may bias the incoming data in ways that are not easy to analyze.  Are
> users who opt in more likely to have a successful install because they pay
> more attention to details as they go?  Less likely, because they are
> tweaking things and will hit other breakage due to hitting uncommon paths?
> More likely, because there's a correlation between willingness to
> participate and willingness to persevere in the face of installer glitches?
> Less likely, because experienced users won't want to participate?

Any means to escape from this will bias the sample, so I think it's not an 
argument for anything except making this impossible to change and I don't 
think anyone wants that.

> The one thing we can say for sure is that if this isn't enabled by default,
> we will miss out on capturing this data for the least engaged subset of
> users, who in a sense are also the most interesting group to get this
> information for:  we can usefully extrapolate from unengaged users as a
> lower bound for installation success, we can't do the converse to ascertain
> anything about the experience of the unengaged users by looking at the
> results from more heavily engaged users.

Up until someone remasters the CD with this preseeded off and starts blogging 
about a version of Ubuntu that doesn't phone home.  Then you don't know 
anything again.

> > This is something that is a significant change that should be reviewed
> > and approved by the Tech Board and possibly the CC.
> 
> I think it's a very good idea to put this question to the TB.

I'm reasonably confident this isn't going to be settled by consensus building, 
so hopefully Evan will do this (I don't think that someone in my position 
really has standing to do this until it's landed).

Scott K