Proposing a New App Developer Upload Process

[Scott Kitterman]

On Tuesday, September 04, 2012 02:32:04 PM David Planella wrote:
> Al 04/09/12 13:06, En/na Scott Kitterman ha escrit:
> > On Monday, September 03, 2012 07:59:15 PM David Planella wrote:
> >> Hi everyone,
> >> 
> >> As many of you will know, in the last few cycles we've been laying out
> >> the foundations to make Ubuntu a target of choice for app developers. I
> >> am not referring to building the platform here (an area in which we also
> >> keep seeking growth), but rather to enabling app authors to create and
> >> distribute original software, to grow a rich ecosystem of independent
> >> apps for our users.
> >> 
> >> The resounding success of our latest initiative, the Ubuntu App
> >> Showdown, has not only shown an explosion of high-quality applications
> >> (created in just 3 weeks!), but more importantly, the growing interest
> >> in developing applications for Ubuntu.
> >> 
> >> As we continue building upon these foundations, we also keep on refining
> >> our processes to identify and improve areas in which we can provide a
> >> better experience for app developers. While doing this, we've been
> >> gathering metrics from different sources –the current queue of apps
> >> pending review and the Showdown results being the main ones. They all
> >> provide clear evidence: the current approach to submit third-party apps
> >> in the Extras repository has already reached the limit in terms of human
> >> review capability and application throughput.
> >> 
> >> Despite our best intentions and the Ubuntu App Review Board's epic
> >> efforts, we're currently putting a strain on reviewers (who cannot keep
> >> up with the incoming stream of apps) and providing an unsatisfactory
> >> experience for app authors (who have to endure long delays to be able to
> >> upload their apps).
> >> 
> >> During the course of the last few weeks, after having identified the key
> >> issues and assessed the different options available, Jono Bacon, Michael
> >> Hall and myself have been working to put forward a proposal for a new,
> >> improved app developer upload process. We've been not alone in this
> >> project: we've had the input and help from many others, including the
> >> App Review Board and members of many other Ubuntu and Canonical teams
> >> (Security, Foundations, Desktop, Consumer Apps, just to name a few).
> >> 
> >> We are now at a point where we consider the spec to be ready for public
> >> review and comment, and we would like to ask for your own contribution
> >> to this process. Ultimately, the goal is to discuss and scope the new
> >> app developer upload process at UDS, but we would like to have a solid
> >> specification to be used in advance as the basis for any UDS sessions
> >> and blueprints. If you'd like to contribute, you can:
> >> 
> >> * Review the spec at https://wiki.ubuntu.com/AppDevUploadProcess
> >> * Provide any feedback either on the Feedback section (preferred) or
> >> 
> >>   continue the discussion on this thread.
> >> 
> >> As the people who create Ubuntu, your input is especially valuable, and
> >> we'd really appreciate it if you could spend some of your time to help
> >> app development in Ubuntu succeed.
> > 
> > I understand the goal of this and why it's the direction those interested
> > in making it easier to provide non-Ubuntu apps want things to go.  It's
> > not at all surprising that the volunteer reviewers were insufficient,
> > since that's always been the case for new Ubuntu packages as well.
> > 
> > I do think the proposal misses a few points though.  The purpose of /opt
> > was not only to limit the access that these apps to the rest of the file
> > hierarchy, it was also to make sure that files provided by these apps
> > could not be in the path of Ubuntu packages and to avoid namespace
> > contamination, particularly in /usr/bin.  This proposal addresses neither
> > of those points, so I think it's inusufficient as it stands.
> 
> Thanks Scott for the feedback. This is indeed a good point and something
> that we've discussed, but I agree that we should better flesh it out in
> the spec.
> 
> However, I disagree in that we're not addressing neither of the points.
> For 1. (limiting access to the rest of the file hierarchy) the proposal
> covers a mechanism for whitelisting a well-known set of file system
> installation locations [1]. It is only additional security in terms of
> installation of system files in the same sense /opt did. Actual access
> to the filesystem is limited through sandboxing, which is the main
> improvement over the /opt approach.
> 
> As per 2. (file system conflicts), we're mentioning:
> 
> "will [...] rely on [...] file path conflict checking on the archive side"
> 
> And I think you're right in that that part needs further development. In
> the past, we've discussed a couple of approaches to detect file system
> conflicts [2]. They were mainly concerned with scanning the file
> contents of packages from different archives, comparing them and
> detecting the conflicts. For the scanning part, we've already got an
> example on packages.ubuntu.com (reads the Contents.gz file from the
> archive and stores the info on a database).
> 
> Any input on these or new alternatives would be very useful.
> 
> Thanks.
> 
> Cheers,
> David.
> 
> [1]
> https://wiki.ubuntu.com/AppDevUploadProcess#TableWhitelistedInstallLocations
> [2] /!\ Note: this document is not related to the current spec and some of
> the information is out of date or just for reference /!\
> https://wiki.ubuntu.com/OptRequirement#Addressing_the_concerns_separately

I'm afraid I wasn't clear about my concerns.

The problem isn't just with file conflicts with current packages, it's that 
these packages will now start using up distro namespace.  If some app 
developer package ships the file /usr/games/bird-game, even though there's no 
current conflict, there is a package sync'ed from Debian that also ships 
/usr/games/bird-game then there's a conflict we have to resolve.  In /opt in a 
proper vendor namespace this can never happen.

As far as I can tell, this is a fundamental problem with the proposal that I 
don't think can be resolved short of installing in /opt (which is why we 
started off doing that).

My concern about files being in the path is not about the access that the app 
developer package has, but that they may inadvertently or maliciously cause 
Ubuntu packages to use files from the app developer package.

A fairly contrived example derived from the above is if the Debian/Ubuntu 
package that shipped /usr/games/bird-game was in the archive and an app 
developer package was uploaded that shipped /usr/bin/bird-game, it could be 
run instead since /usr/bin precedes /usr/games on the standard user path.  Or 
maybe a Python based app ships an embedded copy of a module since they've 
tested with that version and don't want to test with the distro version and 
they write their setup.py to install it in /usr/local/lib/python2.7/dist-
packages.  Suddenly every user of that module is now using the app developer 
package version and not the distro version.

Apparmor doesn't help with this problem.

Once again, if you're installing in /opt, this can't happen and I don't see 
how to solve it other than installing in /opt (which is another reason the 
/opt requirement exists).

The /opt requirement wasn't imposed just to be difficult.  Until all the reasons 
for it are addressed, I think it needs to be maintained.

Scott K