App installer design: click packages

[Thomas Leonard]

On 2013-05-08 11:14, Colin Watson wrote:
> There've been many discussions of late about the challenges of scaling
> app installation to a very large number of apps, including making app
> packages much easier to automatically audit and sandbox, and making them
> safer and quicker to install without the possibility of system-level
> breakage.  This is needed to make it quicker and easier for app
> developers to get their software into Ubuntu.  There are several pieces
> to this (including a good deal of prior discussion in
> https://wiki.ubuntu.com/AppDevUploadProcess), but it seems clear that
> one of those is some kind of simplified packaging format which we can
> offer to third-party application developers.

Hi, 0install developer here. I'd be interested to know more about what 
extra features 0install would need to be useful to you. As Matthias 
said, creating yet another packaging system would be disappointing, 
especially as your goals seem to match ours exactly.

> This is not aimed at changing packages that are already part of the
> Ubuntu archive; for the most part our existing system works well for
> those, and they tend to have non-trivial dependency structures.  We'll
> continue to use dpkg and apt for building the Ubuntu operating system,
> syncing with Debian, and so on.  There's no point developing a packaging
> system for apps and making it have the full panoply of features needed
> for the Ubuntu archive: it'd just be second-system-effect on top of our
> current packaging system.  So the scope of what I've been considering is
> purely leaf apps built on a fixed "base system", which in the case of
> the initial target of the Ubuntu phone/tablet work would be the run-time
> part of the Ubuntu SDK.  The radically-reduced dependency structure
> means that most apps will be Ubuntu-SDK-specific to start with, although
> I'd like to make sure that the package format design includes enough
> support up-front to allow this to be useful for other platforms that
> define suitable base system profiles in future.
>
> So, at Steve Langasek's request, I've been putting together a proof of
> concept of a low-level app package installer and packaging format.
> Highlights of what it can do so far are:
>
>   * no dependencies between apps; single implicit dependency on the base
>     system by way of a Click-Base-System field

0install packages can depend on debs if they want to, so if you create a 
virtual deb package representing the Ubuntu base system then you could 
just depend on that (e.g. "requires Ubuntu >= 13.4"). If you want to be 
cross platform, it would make more sense to specify individual 
dependencies, though.

>   * installs each app to an entirely separate directory

Yes.

>   * entirely declarative: maintainer scripts are forbidden

Yes.

>   * base package manager overhead, i.e. the time required to install a
>     trivial package containing a single small file, is about 0.15 seconds
>     on a newish x86 laptop and about 0.6 seconds on a Nexus 7 (and that's
>     with the current prototype implementation in Python; a later
>     implementation could be in C and would then be faster still)

0.109 seconds here on my laptop (including the time to download the 
archive from Apache running on localhost; normally network delays will 
dominate, of course).

>   * not limited to installing as root, although there may be similar
>     constraints elsewhere to ensure that apps can't edit their own code
>     at run-time

Same. If system-wide sharing is enabled then downloads go under 
/var/cache and so apps cannot edit their own code. Without system-wide 
sharing, the files are stored in the user's home directory. They're 
read-only, but Linux doesn't currently provide us with a way to stop an 
app running as a user from changing that user's files. We're looking 
into using the new user namespaces support for sandboxing.

>   * packages built by feeding the intended output directory tree to a
>     simple Python tool, plus a manifest.json file

Same, but the metadata is XML, not JSON.

>   * building packages requires only the Python standard library, with the
>     intent that it should be possible to build these packages quite
>     easily on non-Ubuntu or even non-Linux systems

Same. 0install packages can run across Linux, OS X, Windows, Unix, etc 
(assuming the packaged software can too).

>   * binary packaging format sufficiently similar to existing one that we
>     could add support to higher-level tools with minimal effort

Package format is .tar.bz2 (other similar compressed archive formats, 
such as .zip are also supported).

>   * strawman design for hooks into system packages, which will be
>     entirely declarative from the app's point of view

What does this mean?

>   * unit-tested from the start

Of course.

> Obvious items I still need to work on:
>
>   * produce a strawman hooks implementation with some real worked
>     examples
>   * integrate (or demonstrate how to integrate) the container isolation
>     properties worked on elsewhere

Would be interested to hear more about this.

>   * Click-Base-System field is skeletally simple right now, and may need
>     to be expanded to at least leave open the possibility of multiple
>     flavours of base system (see also GNOME's profiles idea)
>   * adjust unpack handling to avoid problems with project renames and
>     name clashes, and to unpack each version into its own directory and
>     flip symlinks to allow for multi-user independence
>   * integrate into the Ubuntu SDK, as well as providing examples of how
>     it can be integrated into other build systems too
>
> To do, but not in this project:
>
>   * package acquisition
>   * indexing support and other frontend things (i.e. the app store)
>   * upload handling
>
> So far I've demonstrated to my own satisfaction that we can get
> acceptable performance and isolation characteristics with very little
> work; the installer part on its own is under 300 lines of Python right
> now, which I consider useful since bugginess is often proportional to
> code size.  I should have the prototype ready for people to look at in
> time for UDS next week, and I'll ensure that there's a session scheduled
> for this.
>
> Is there anything else people can think of that a system like this needs
> to consider?

Package signing, key distribution and trust, mirroring, dependencies, 
roll-back, parallel installation of stable and testing versions, 
checking for updates, source packages, ...? ;-)

> Footnote: Why not use one of the existing systems out there?
>
> We still might, but at present it's not clear that it would make a whole
> lot of difference.
>
> Many of the existing app packaging systems are more of a reflection of
> the system they were built for than anything else.  If you look at, say,
> Android's APK format, it's essentially a zip file with a manifest and
> some conventions about Dalvik class installation and the like.  Most of
> the other mobile app formats are similar.  Things like Listaller might
> be a reasonable fit, but I'm worried about importing things like a full
> dependency solver into this: much though I love dependency-based
> packaging systems, they necessarily involve scanning the system
> packaging database at some point and I would much prefer app packages to
> be as independent of that as possible, mainly for performance but also
> to be as bulletproof as possible.  GNOME app bundles appear to have a
> rather different sandboxing model than we need (e.g. enforced
> restrictions on dependencies), we probably want better system
> integration, and perhaps other differences; but that said I like the
> coarse-grained profile model of dependencies (and unknowingly imitated
> it), and it might make sense to share some profile names.

 > Barry spent
> some time looking at 0install, and it wasn't too bad a fit but we would
> still need to solve many of the same system integration problems.

Is there a list of these issues somewhere?

> The proof of concept I wrote also isn't entirely new code.  It's tiny
> due to using .deb as a container format (minus maintainer scripts, full
> dependencies, etc.), so I get to save effort by using dpkg to unpack
> things, which leaves us room to selectively use more of its features in
> future if we want to.  This is quite a compelling approach to me for
> various reasons.  I could be persuaded to look at something else, but I
> don't think there's actually much code to share at the packaging format
> level anyway, as it's so simple; quite possibly more at the sandboxing
> level, but that's really a separate layer.
>
> Furthermore, the priority for this system at present is for Ubuntu
> phone/tablet app packages, which haven't actually been built yet.  As a
> result, I'm less worried about fragmentation than I might otherwise be.
> I would like to make sure that it's usable elsewhere too, although
> somebody would need to work out what that means in terms of base system
> profiles.
>
> Although I'm very familiar with dpkg and traditional package management,
> I only started looking at this side of things a couple of weeks ago, so
> I'm sure I'm behind on research.  Pointers welcome.


Regards,


-- 
Dr Thomas Leonard        http://0install.net/
GPG: 9242 9807 C985 3C07 44A6  8B9A AE07 8280 59A5 3CC1
GPG: DA98 25AE CAD0 8975 7CDA  BD8E 0713 3F96 CA74 D8BA