App installer design: click packages

[Thomas Leonard]

On 2013-05-14 14:37, Colin Watson wrote:> On Thu, May 09, 2013 at 
09:41:26AM +0100, Thomas Leonard wrote:
 >> On 2013-05-08 11:14, Colin Watson wrote:
 >>> So, at Steve Langasek's request, I've been putting together a proof of
 >>> concept of a low-level app package installer and packaging format.
 >>> Highlights of what it can do so far are:
 >>>
 >>>   * no dependencies between apps; single implicit dependency on the 
base
 >>>     system by way of a Click-Base-System field
 >>
 >> 0install packages can depend on debs if they want to, so if you
 >> create a virtual deb package representing the Ubuntu base system
 >> then you could just depend on that (e.g. "requires Ubuntu >= 13.4").
 >> If you want to be cross platform, it would make more sense to
 >> specify individual dependencies, though.
 >
 > This is a deliberate constraint to avoid interaction with the (large)
 > system package database, rather than a feature that was too hard to
 > implement.

To avoid speed problems, when 0install queries the dpkg database, it 
caches the results here:

   ~/.cache/0install.net/injector/dpkg-status.cache

This is a text file with lines of the form "name=version<Tab>arch".
It invalidates the cache if the mtime of /var/lib/dpkg/status changes.

In general, 0install should be fast as long as the dpkg database isn't 
changing frequently (and if it is, those updates will be causing far 
more slowdown than 0install's querying of it on the next app update).

 > Is it easy to disable this feature in 0install and do
 > something more like the system that Listaller and click-package use of
 > depending on system frameworks declared in text files?

The code for distribution-specific packages is in 
zeroinstall/injector/distro.py. Change get_host_distribution() to return 
Distribution() rather than DebianDistribution() if you want to 
completely ignore distribution packages. Of course, that means that 
existing packages won't work unless all their (declared) dependencies as 
available as 0install packages too.

If a 0install package doesn't depend on anything which could be provided 
by the distribution, then dpkg will never be queried. However, many 
existing programs do depend on such things. For example, Python and Java 
are usually provided by distribution packages, so all existing Python 
and Java programs will cause a check of whether their VM is available 
natively.

You could provide an alternative to DebianDistribution that gets its 
information from a text file, but you'd have to update the text file 
somehow based on the dpkg database, which is basically what 0install is 
doing anyway.

 > (I suppose it's not critical; the primary requirement is bulletproof
 > reliability, and performance is merely an important secondary
 > requirement.)
 >
 >>>   * base package manager overhead, i.e. the time required to install a
 >>>     trivial package containing a single small file, is about 0.15 
seconds
 >>>     on a newish x86 laptop and about 0.6 seconds on a Nexus 7 (and 
that's
 >>>     with the current prototype implementation in Python; a later
 >>>     implementation could be in C and would then be faster still)
 >>
 >> 0.109 seconds here on my laptop (including the time to download the
 >> archive from Apache running on localhost; normally network delays
 >> will dominate, of course).
 >
 > debs install pretty quickly on my laptop too, but querying the package
 > database is going to be slow on typical phone hardware, so I'd still
 > like to make sure we avoid that.
 >
 >>>   * not limited to installing as root, although there may be similar
 >>>     constraints elsewhere to ensure that apps can't edit their own code
 >>>     at run-time
 >>
 >> Same. If system-wide sharing is enabled then downloads go under
 >> /var/cache and so apps cannot edit their own code. Without
 >> system-wide sharing, the files are stored in the user's home
 >> directory. They're read-only, but Linux doesn't currently provide us
 >> with a way to stop an app running as a user from changing that
 >> user's files. We're looking into using the new user namespaces
 >> support for sandboxing.
 >
 > I think we will have some degree of sharing but not to the extent where
 > one user can force another to upgrade, so we'll be running the installer
 > as a separate user.

We share the package data, but not the metadata saying which version is 
current. So each user updates on their own schedule, but when it comes 
to download time, they may find the version they want is already in the 
system cache because some other user downloaded it already.

Files in the system cache are owned by the "zeroinst" user. We don't do 
any downloading of files as the "zeroinst" user (we used to, but it 
increases the attack surface a lot and operations such as using mirrors 
and cancelling shared downloads are tricky). Instead, the user's 
0install process downloads the package itself and passes it to 
"zeroinst", which verifies that its contents match its name (a secure 
hash) and stores it under that hash (e.g. as 
/var/cache/0install.net/implementations/sha256=e4f835.../).

 >>>   * strawman design for hooks into system packages, which will be
 >>>     entirely declarative from the app's point of view
 >>
 >> What does this mean?
 >
 > You mean "entirely declarative from the app's point of view"?  I mean
 > that the app contains no code executed with the permissions of the
 > package installer, although system packages may contain code that is
 > executed with those permissions when apps install certain kinds of files
 > or declare certain kinds of metadata.
 >
 >>> Obvious items I still need to work on:
 >>>
 >>>   * produce a strawman hooks implementation with some real worked
 >>>     examples
 >>>   * integrate (or demonstrate how to integrate) the container isolation
 >>>     properties worked on elsewhere
 >>
 >> Would be interested to hear more about this.
 >
 > https://click-package.readthedocs.org/en/latest/hooks.html is the short
 > answer for now.

Seems reasonably, though I'd probably have the hooks in a fixed "hooks" 
location and change the consuming service to look there, rather than 
letting the package specify a pathname of its choice and then, 
presumably, filtering it against some whitelist. Saves having to 
maintain the whitelist.

Also, I'd name the hook after the app's "petname" (a unique name chosen 
by the user) rather than its URI, since 0install allows multiple 
installations of the same software. e.g. a user might have "firefox" 
(their default browser), "firefox-banking" (with no plugins) and 
"firefox-dev" (a newer version for testing), but you might not care 
about that.

 >>> Is there anything else people can think of that a system like this 
needs
 >>> to consider?
 >>
 >> Package signing, key distribution and trust, mirroring,
 >
 > All of these belong in a higher-level tool.
 >
 >> dependencies,
 >
 > Intentionally excluded.
 >
 >> roll-back, parallel installation of stable and testing
 >> versions,
 >
 > Different versions are unpacked into different directories, so these are
 > easy.  (In fact the main thing to do is the converse: decide when to
 > garbage-collect old versions.)
 >
 >> checking for updates,
 >
 > Higher-level tool.
 >
 >> source packages, ...? ;-)
 >
 > I made a conscious choice not to do anything like Debian source
 > packages.  I have no problem with them personally, but they're a
 > consistent source of confusion among app developers, and of course (ugh)
 > there is the inevitable confusion for distributors of proprietary apps.
 > However, I'm conscious that it's important to make it easy for people to
 > distribute free software the right way.  This is an open question for
 > now and I've put it on the to-do list.
 >
 >>> Barry spent
 >>> some time looking at 0install, and it wasn't too bad a fit but we would
 >>> still need to solve many of the same system integration problems.
 >>
 >> Is there a list of these issues somewhere?
 >
 > Barry Warsaw would have to speak to that, I think.  But mainly, there's
 > no way for the system to declare any kind of integration facilities at
 > all, as far as I can see, so we'd have to solve that problem anyway.
 >
 > I will certainly concede that the rest of 0install is very flexible and
 > has a usefully pluggable design; you seem to have done a good job on it
 > and Barry spoke well of it.  It just seems that there's a bit of a
 > mismatch in that it's kind of over the top in terms of flexibility for
 > what we need and seems to be entirely missing some other things we need,
 > as above; and it would be a lot of code to rewrite in some other
 > language if (as is possible, though not imminent) we decide that Python
 > is just too heavyweight to run routinely on the Ubuntu phone.
 >
 > Thanks,
 >