App installer design: click packages

[Matthias Klumpp]

Hi!

2013/5/9 Colin Watson <cjwatson at ubuntu.com>:
> On Wed, May 08, 2013 at 03:25:42PM +0200, Matthias Klumpp wrote:
>> Disclaimer: As developer of Listaller, AppStream and PackageKit, I am
>> of course biased on this topic ;-)
>
> Thanks for your mail.  I'm looking over Listaller at the moment, and
> expect to have some more comments once I've done that.  However, I'm
> going to be mostly offline from Friday to Sunday, so a partial reply in
> the meantime:
Great! Please keep in mind that the project will undergo a transition
to greatly simplyify dependency solving soon, with an option to
restore and improve the 0install compatibility too. (there has been
discussion about this with 0install people some time ago, but we did
not find a working solution back then).
The key values for Listaller are distribution-independence and
integration (= everything which supports PackageKit can handle
Listaller stuff too).

>> 2013/5/8 Colin Watson <cjwatson at ubuntu.com>:
>> >  * not limited to installing as root, although there may be similar
>> >    constraints elsewhere to ensure that apps can't edit their own code
>> >    at run-time
>>
>> Non-root installs into user's $HOME are implemented & working in
>> Listaller, but due to the tight integration into PackageKit, this
>> feature should not be used.
>
> In practice, I don't actually intend to install packages as the ordinary
> calling user directly; as I indicated above, this would be a bad idea
> because it would permit apps to overwrite their own code.  However, it's
> convenient for testing, and it may make sense to unpack apps as a
> special "software" user or similar to reduce the system's exposure to
> things like vulnerabilities in underlying unarchiving tools, which are
> rare but not unknown.  (Or we could just hit it with AppArmor.  I don't
> much care either way as long as we avoid the obvious flaw.)
Yes, that is the rationale for allowing things like this :-) If there
is a good way to lock 3rd-party applications with AppArmor, having a
feature like this would definitely help. But maybe the sandbox will
already provide enough security.

>> >  * building packages requires only the Python standard library, with the
>> >    intent that it should be possible to build these packages quite
>> >    easily on non-Ubuntu or even non-Linux systems
>>
>> Hmm, why? If a non-Python application should be packaged, this feature
>> would be lost, as people will have to crossbuild it anyway. And in
>> order to build stuff for the upcoming Ubuntu products, people will
>> need to install it (at least in a VM) anyway, for testing purposes.
>
> Remember that the main target of our project, at least for the time
> being, is Ubuntu Touch - phone/tablet apps.  We expect a significant
> percentage of these to be written in pure QML or HTML5/CSS, and to make
> it possible to carry these over to desktop apps in time as well.  I
> think it's important to minimise the barrier to entry to creating such
> apps; if somebody wants to create them on other systems they should be
> able to.
For all QML/HTML5 stuff, this indeed makes sense. But since packages
are not magic at all (often just compressed tarballs with some meta
information), implementing that is easily possible. 0install proves
that it even works for deploying stuff on MacOS, Windows and Linux.

>> >  * binary packaging format sufficiently similar to existing one that we
>> >    could add support to higher-level tools with minimal effort
>> >  * strawman design for hooks into system packages, which will be
>> >    entirely declarative from the app's point of view
>>
>> Is this going to be Ubuntu-specific? If not, you will have some
>> trouble with that...
>
> I wasn't planning to put "ubuntu" in the hook paths or anything, but it
> is indeed up to the system to declare how it wants apps to integrate
> with it.  I don't see a sensible way around that; it is entirely wrong
> to put system/app integration code in the installer itself, and any such
> approach would result in problems with the system and the installer
> getting out of sync further down the line.
You mean the package-information with "installer"? Or do you mean the
application which installs things? The package itself should not have
any assumptions on how it will be installed on the system, and how the
system would look like. The application which installs the package,
however, is responsible for integrating the new application into the
system, and also for negotiating with the native package manager about
installed requirements.
Both Listaller and 0install force applications to be developed in a
way that they don't have any fixed assumptions on the system they are
installed on. Listaller even contains many tools due to it's partial
Autopackage origin which help to minimize and lower system
dependencies.
All of these features don't matter much if you want to use a 3rd-party
package manager just on the Ubuntu Phone, because I assume that the
phone software will have some kind of API/Platform level applications
will be built for. But if you want to target the desktop too, you will
need an installer which can handle the heterogeneous "Linux Platform"
(e.g. the case if you install a GNOME application, but run KDE).

> dpkg is right to defer all integration code to packages; the hook scheme
> I'm considering is mainly inspired by dpkg triggers.  (We may or may not
> be able to call dpkg triggers directly; for performance it is vital to
> avoid having to actually open the dpkg database.)
I wonder which triggers 3rd-party apps could possibly need... If e.g.
new .desktop files are installed, the installer can take action and
update the desktop database, same if a manual page is installed, so no
need for having the package declare an explicit trigger.
Which kind of triggers do you think would be needed?

>> > Footnote: Why not use one of the existing systems out there?
>> >
>> > We still might, but at present it's not clear that it would make a whole
>> > lot of difference.
>>
>> If it doesn't, why don't you use them? :P
>
> It's a trade-off between the benefit of using an existing system and the
> cost of twisting it around to fit what we actually need it to do.  I'm
> not fundamentally opposed to using an existing system - I certainly
> understand why people want us to, and that there's a perception that
> Canonical unnecessarily does its own thing (you can look up my own
> commit record if you like) - but if there's a mismatch between our goals
> then it may end up being rather more trouble than it's worth.
Yes, this makes some sense - but it would be very natural in this case
to contact people from Listaller, 0install and GNOME directly, explain
your goals and ask them if their software would match these goals.
Upstream projects might even adjust the project code to match
Canonical's needs too.

> One thing
> I'm particularly concerned about is that a system that has tried to be
> very general may end up reinventing similar scaling and reliability
> problems as the packaging system we already have.
Hehe, yes. These tendencies have been seen in Listaller's code some
time ago, which resulted in a review of the dependency-resolving
plans. The problems were mainly because many distributions did not
provide adequate metadata for Listaller to use, so dependency colving
was overly complicated *and* not working great on all distributions.
This situation already became better, and the issues will be gone
completely with the next release (at the cost of breaking the IPK
package format slightly, which is okay at the current point in time.).
As for enhanced metadata, I am working on this for Debian (and maybe
Ubuntu will adopt the changes?), but this is a different topic (and
won't be needed anymore for Listaller then).

> (To be clear - hi Slashdot - I'm very fond of the dpkg/apt system and
> have put a great deal of effort into it over the last decade or so, both
> on my own time and for pay; I think it's excellent and have no desire to
> kill it.  But I also recognise that it offers all the rope you need to
> hang yourself with, and its failure modes in the face of input from
> inexperienced developers are sometimes a little on the abrupt side.)
>
> It's also worth noting that I have had roughly one response for each of
> several alternative projects so far; the field is clearly already highly
> fragmented.
I got no mail ;-)
As for the field, there is 0install, Listaller and LOKI/Mojo. Also the
GNOME stuff / app-bundle stuff in form of Glick2 and Klik-like tools.
It's not *that* fragmented: LOKI/Mojo is mainly used for games, and
these packages are binary applications, which contain both the logic
and data for installation. Something I would like to kill with fire. I
spoke with Ryan C. Gordon about it a long time ago, and there is a
chance for replacing it.
The GNOME Glick2 stuff is still in some kind of planning phase -
unfortunately I have been unable to retrieve information about the
plans yet (missed the panel at FOSDEM), but these things are a very
general approach on the installation issue (covering advanced
privileges and DBus-in-kernel as parts of it). The current state can
merely be seen as a technical preview.
IMHO the approach of having app-bundles is flawed, mainly because of
data-duplication and missing upgradeability (but not everyone things
like that).
0install has a very large scope, covering installations on MacOS,
Windows and Linux using a nice feed format and a good dependency
solver. Some projects use it with success.
Listaller was rewritten in Vala some time ago, and merged with the
Autopackage codebase. It is Linux-only (but might also work on *BSD)
and has integration as key-feature. It hooks into the PackageKit
interfaces, making e.g. updates of 3rd-party applications available in
the system's updater application, along with all other system updates.
It is intregrated into Apper (KDE-PackageKit-package manager), support
for GNOME is currently in progress. The API is unstable until version
0.6 is shipped.
And there is also Nix, a full-fledged package manager, featuring
atomic upgrades and complete rollbacks, as well as parallel
installation of multiple package versions. I have not yet looked too
close at it, but it seems to be working well.

>> > Many of the existing app packaging systems are more of a reflection of
>> > the system they were built for than anything else.  If you look at, say,
>> > Android's APK format, it's essentially a zip file with a manifest and
>> > some conventions about Dalvik class installation and the like.  Most of
>> > the other mobile app formats are similar.  Things like Listaller might
>> > be a reasonable fit, but I'm worried about importing things like a full
>> > dependency solver into this: much though I love dependency-based
>> > packaging systems, they necessarily involve scanning the system
>> > packaging database at some point and I would much prefer app packages to
>> > be as independent of that as possible, mainly for performance but also
>> > to be as bulletproof as possible.
>>
>> That is the plan :-) The current Listaller dependency-solving is in a
>> bad shape, because it relies on the native package management, and
>> only RPM-based distros can deliver all required information for
>> Listaller to be able to solve dependencies.
>
> Is this because you're using file dependencies or something?  In most
> other respects dpkg is at least as capable as RPM.
Yes, file dependencies/library dependencies is the problem :) As for
libraries, dpkg can deliver the required data due to the very strict
package-naming scheme for library-packages. But that's still not
optimal. This resuled in Listaller working much better on Fedora than
on Debian (on Debian, it was always falling back to other dependency
sources, instead of installing native packages). The plans for
Listaller will result in less-flexible dependency solving, but will
make it work the same way on most distributions.

>> The plan for the next release (which was defered, it should have been
>> available in the previous release, but didn't make it), is to really
>> simplify dependency solving. This is done by defining "Frameworks",
>> which is e.g. KPlatform5, GNOME3, etc. which have to be installed
>> using the system package manager (Listaller will fail if no sufficient
>> version of them is present). These frameworks are defined through
>> small textfiles, which describe the items which have to be present to
>> consider a framework as available. Distributors and authors of the
>> upstream software will have to write these files.
>
> Right, this is equivalent to the skeletal "Click-Base-System" notion,
> and I was also expecting to use the presence of a simple text file to
> define that.  (I'm not wedded to the name; I've seen both "framework"
> and "profile" used to describe this, and either is probably a better
> name.  I'd be fine with adopting an existing standard here.)
There is no standard ;-) AFAIK GNOME and KDE call it framework, and
the name made sense, that's why it is used.

>> The second aspect is "Modules", shared software which depends on
>> Frameworks and which Listaller can try to satisfy using the native
>> package manager.
>> People should, if possible, only use Frameworks, but for some
>> packages, being able to depend on a shared module seems to be
>> desirable (e.g. the OggVorbis libraries can easily be satisfied via
>> Listaller).
>> Frameworks support will definitively be available in the next release,
>> Modules might be added later.
>
> This specifically concerns me, and is one reason I'm sceptical about
> using an existing system that might have this kind of feature or be
> tempted to add it for the sake of being more complete.  It means that as
> soon as any app decides to use a module, you suddenly have to open the
> system package manager, and deal with a large percentage of the
> associated problems (for example, you need to deal with ABI changes in
> the additional system packages).
If there is an ABI change done properly, the installer will recognize
a changed SONAME and refuse to install the package which requires a
different one, or it might use some logic to satisfy that dependency
anyway (by downloading it from somewhere else).
As for Ubuntu packages, you could disallow the use of modules for the
disribution, and only allow the Frameworks subset. So application
authors will only use these (and packages requiring unsatisfied
modules will simply not be installed).
That is one advantage of separating these components. I created some
diagrams about that a year ago (need to look where these are now...) -
a bit bad that these ideas are not yet completely realized.

> I realise that people are concerned about bloat in apps.  I would say
> that we need to deal with this by making the Ubuntu SDK sufficiently
> capable to do what people actually need on phones and tablets.  For
> later desktop convergence, I do want this system to *work*, but I don't
> think it's a priority to make it meet the requirements of everything
> people might want to write.  One package might want Vorbis; one might
> deliver an Emacs addon; one might rely on a complex and unusual language
> framework with lots of loadable modules.  Where do you draw the line?
Everything which needs tight system integration (= installs a
PolicyKit privilege) shouldn't be installed using 3rd-party packages,
but instead be packaged in a native package. For scripting languages,
a mechanism to load dependencies can be offered (e.g. for Python
modules).
In general, the line is defined through the availability of
Frameworks/Modules on the system.
If there is no *.framework file for something an application is
requiring, there is no way to install it. Period. For Modules, the
system itself defines which ones can be installed, if there is no way
to satisfy them, the application will also not be installed.

> If you try to make everything available, you are rather likely to end up
> reinventing the system package manager and gaining little in the
> process.  My preference is to draw the line quite conservatively, at the
> SDK which we document on developer.ubuntu.com for app developers, and
> not to worry if it doesn't currently meet all the needs we might have in
> the future; we can always extend the SDK, or with something like the
> frameworks-declared-in-text-files idea you mentioned other people can
> declare their own.
This is sane.

>> > The proof of concept I wrote also isn't entirely new code.  It's tiny
>> > due to using .deb as a container format (minus maintainer scripts, full
>> > dependencies, etc.), so I get to save effort by using dpkg to unpack
>> > things, which leaves us room to selectively use more of its features in
>> > future if we want to.  This is quite a compelling approach to me for
>> > various reasons.  I could be persuaded to look at something else, but I
>> > don't think there's actually much code to share at the packaging format
>> > level anyway, as it's so simple; quite possibly more at the sandboxing
>> > level, but that's really a separate layer.
>>
>> Hmm... This will make it very difficult for other distributions to use.
>
> I don't think so, really.  dpkg is straightforward to build elsewhere,
> so it's no worse than LSB packages, which were RPMs; actually easier
> since in this case you don't need to either have a system dpkg database
> or fake it by converting the package into some other format first (e.g.
> alien).  Yes, LSB packaging wasn't exactly a raging success, but it
> failed much more due to lack of interest than due to problems with the
> packaging format.  (LSB compatibility has been either available by
> default or very easily installable in Ubuntu since the project started.)
It also failed because LSB-RPM dependencies cannot be easily solved on
non-RPM systems. But yes, making a package format with the features
you outlines available on more distributions would be much easier.

> I think most ordinary users would have to look up what to do with a
> Listaller IPK if they saw one and would still have to go and install
> some software first to make use of it, so this seems like a neutral
> point to me.
Ideally, they simple install it using GNOME-PackageKit or Apper, just
like a native package ;-) But additional software is still needed,
unless Listaller is shipped by default. Autopackage did it
differently, but I am happy that we don't have binary formats anymore.
:)

Cheers,
    Matthias