App installer design: click packages

[Matthias Klumpp]

Hi!
Disclaimer: As developer of Listaller, AppStream and PackageKit, I am
of course biased on this topic ;-)
I find it very sad that so many parties currently start reinventing
the basic ideas of Listaller, instead of helping the Listaller project
to improve, and kill the remaining flaws.

2013/5/8 Colin Watson <cjwatson at ubuntu.com>:
> There've been many discussions of late about the challenges of scaling
> app installation to a very large number of apps, including making app
> packages much easier to automatically audit and sandbox, and making them
> safer and quicker to install without the possibility of system-level
> breakage.  This is needed to make it quicker and easier for app
> developers to get their software into Ubuntu.  There are several pieces
> to this (including a good deal of prior discussion in
> https://wiki.ubuntu.com/AppDevUploadProcess), but it seems clear that
> one of those is some kind of simplified packaging format which we can
> offer to third-party application developers.
This is the rationale of Listaller, see http://listaller.tenstral.net/

> So, at Steve Langasek's request, I've been putting together a proof of
> concept of a low-level app package installer and packaging format.
> Highlights of what it can do so far are:
>
>  * no dependencies between apps; single implicit dependency on the base
>    system by way of a Click-Base-System field
Implemented in Listaller, but poorly working. The dependency-solver is
currently being rewritten, with the upcoming KDE-Frameworks 5 and
GNOME-Platform (aka "GNOME-OS") in mind.

>  * installs each app to an entirely separate directory
Done in Listaller, but several shared resources can - well - easily be shared.

>  * entirely declarative: maintainer scripts are forbidden
Design decision for Listaller (resulted in controversial discussions,
but finally everyone seems to be fine with it).

>  * base package manager overhead, i.e. the time required to install a
>    trivial package containing a single small file, is about 0.15 seconds
>    on a newish x86 laptop and about 0.6 seconds on a Nexus 7 (and that's
>    with the current prototype implementation in Python; a later
>    implementation could be in C and would then be faster still)
I did not (yet) do measurements on this, but as Listaller was
rewritten in Vala some time ago, I expect it to be reasonably fast.

>  * not limited to installing as root, although there may be similar
>    constraints elsewhere to ensure that apps can't edit their own code
>    at run-time
Non-root installs into user's $HOME are implemented & working in
Listaller, but due to the tight integration into PackageKit, this
feature should not be used.

>  * packages built by feeding the intended output directory tree to a
>    simple Python tool, plus a manifest.json file
Listaller uses a "depscan" tool to auto-detect dependencies. Currently
only works for binary shared libraries, due to lack of time and
Python/Ruby/Perl knowledge to make a sane implementation for these
languages. But the tool is designed with that in mind. Output is a
textfile in Debian-control-file style.

>  * building packages requires only the Python standard library, with the
>    intent that it should be possible to build these packages quite
>    easily on non-Ubuntu or even non-Linux systems
Hmm, why? If a non-Python application should be packaged, this feature
would be lost, as people will have to crossbuild it anyway. And in
order to build stuff for the upcoming Ubuntu products, people will
need to install it (at least in a VM) anyway, for testing purposes.

>  * binary packaging format sufficiently similar to existing one that we
>    could add support to higher-level tools with minimal effort
>  * strawman design for hooks into system packages, which will be
>    entirely declarative from the app's point of view
Is this going to be Ubuntu-specific? If not, you will have some
trouble with that...

Listaller also offers update-sources for installed 3rd-party apps, a
way to build binary apps using older library symbols and some
cross-distro compatibility stuff, a tight integration with PackageKit
(that might actually be a problem for Ubuntu...), which allows updates
for these packages to be displayed in standard package management
tools, alongside other software which was deployed using the system
package manager.

> Footnote: Why not use one of the existing systems out there?
>
> We still might, but at present it's not clear that it would make a whole
> lot of difference.
If it doesn't, why don't you use them? :P

> Many of the existing app packaging systems are more of a reflection of
> the system they were built for than anything else.  If you look at, say,
> Android's APK format, it's essentially a zip file with a manifest and
> some conventions about Dalvik class installation and the like.  Most of
> the other mobile app formats are similar.  Things like Listaller might
> be a reasonable fit, but I'm worried about importing things like a full
> dependency solver into this: much though I love dependency-based
> packaging systems, they necessarily involve scanning the system
> packaging database at some point and I would much prefer app packages to
> be as independent of that as possible, mainly for performance but also
> to be as bulletproof as possible.
That is the plan :-) The current Listaller dependency-solving is in a
bad shape, because it relies on the native package management, and
only RPM-based distros can deliver all required information for
Listaller to be able to solve dependencies.
The plan for the next release (which was defered, it should have been
available in the previous release, but didn't make it), is to really
simplify dependency solving. This is done by defining "Frameworks",
which is e.g. KPlatform5, GNOME3, etc. which have to be installed
using the system package manager (Listaller will fail if no sufficient
version of them is present). These frameworks are defined through
small textfiles, which describe the items which have to be present to
consider a framework as available. Distributors and authors of the
upstream software will have to write these files.
The second aspect is "Modules", shared software which depends on
Frameworks and which Listaller can try to satisfy using the native
package manager.
People should, if possible, only use Frameworks, but for some
packages, being able to depend on a shared module seems to be
desirable (e.g. the OggVorbis libraries can easily be satisfied via
Listaller).
Frameworks support will definitively be available in the next release,
Modules might be added later.

> GNOME app bundles appear to have a
> rather different sandboxing model than we need (e.g. enforced
> restrictions on dependencies), we probably want better system
> integration, and perhaps other differences; but that said I like the
> coarse-grained profile model of dependencies (and unknowingly imitated
> it), and it might make sense to share some profile names.
The thing which worries me a lot is upgradeability of these
app-bundles. Application bundles also suffer from some other issues,
but they have the advantage to be entirely self-contained.

> Barry spent
> some time looking at 0install, and it wasn't too bad a fit but we would
> still need to solve many of the same system integration problems.
Yup, although I think the 0install people would help with that :-)

> The proof of concept I wrote also isn't entirely new code.  It's tiny
> due to using .deb as a container format (minus maintainer scripts, full
> dependencies, etc.), so I get to save effort by using dpkg to unpack
> things, which leaves us room to selectively use more of its features in
> future if we want to.  This is quite a compelling approach to me for
> various reasons.  I could be persuaded to look at something else, but I
> don't think there's actually much code to share at the packaging format
> level anyway, as it's so simple; quite possibly more at the sandboxing
> level, but that's really a separate layer.
Hmm... This will make it very difficult for other distributions to use.

> Furthermore, the priority for this system at present is for Ubuntu
> phone/tablet app packages, which haven't actually been built yet.  As a
> result, I'm less worried about fragmentation than I might otherwise be.
> I would like to make sure that it's usable elsewhere too, although
> somebody would need to work out what that means in terms of base system
> profiles.
>
> Although I'm very familiar with dpkg and traditional package management,
> I only started looking at this side of things a couple of weeks ago, so
> I'm sure I'm behind on research.  Pointers welcome.
Hehe :) It would be cool to stay in contact about this, even if you
decide to create a new and own solution!

Regards,
    Matthias
-- 
Debian Developer | Freedesktop-Developer
KDE-Developer    | GNOME-Contributor
I welcome VSRE emails. See http://vsre.info/