App installer design: click packages

Colin Watson cjwatson at ubuntu.com
Thu May 9 12:15:47 UTC 2013


On Wed, May 08, 2013 at 03:25:42PM +0200, Matthias Klumpp wrote:
> Disclaimer: As developer of Listaller, AppStream and PackageKit, I am
> of course biased on this topic ;-)

Thanks for your mail.  I'm looking over Listaller at the moment, and
expect to have some more comments once I've done that.  However, I'm
going to be mostly offline from Friday to Sunday, so a partial reply in
the meantime:

> I find it very sad that so many parties currently start reinventing
> the basic ideas of Listaller, instead of helping the Listaller project
> to improve, and kill the remaining flaws.

I would prefer to keep the discussion technical rather than emotional.

> 2013/5/8 Colin Watson <cjwatson at ubuntu.com>:
> >  * not limited to installing as root, although there may be similar
> >    constraints elsewhere to ensure that apps can't edit their own code
> >    at run-time
> 
> Non-root installs into user's $HOME are implemented & working in
> Listaller, but due to the tight integration into PackageKit, this
> feature should not be used.

In practice, I don't actually intend to install packages as the ordinary
calling user directly; as I indicated above, this would be a bad idea
because it would permit apps to overwrite their own code.  However, it's
convenient for testing, and it may make sense to unpack apps as a
special "software" user or similar to reduce the system's exposure to
things like vulnerabilities in underlying unarchiving tools, which are
rare but not unknown.  (Or we could just hit it with AppArmor.  I don't
much care either way as long as we avoid the obvious flaw.)

> >  * building packages requires only the Python standard library, with the
> >    intent that it should be possible to build these packages quite
> >    easily on non-Ubuntu or even non-Linux systems
> 
> Hmm, why? If a non-Python application should be packaged, this feature
> would be lost, as people will have to crossbuild it anyway. And in
> order to build stuff for the upcoming Ubuntu products, people will
> need to install it (at least in a VM) anyway, for testing purposes.

Remember that the main target of our project, at least for the time
being, is Ubuntu Touch - phone/tablet apps.  We expect a significant
percentage of these to be written in pure QML or HTML5/CSS, and to make
it possible to carry these over to desktop apps in time as well.  I
think it's important to minimise the barrier to entry to creating such
apps; if somebody wants to create them on other systems they should be
able to.

Running a separate test device may well be an easier proposition than
building on it.  For example, you might have a physical device (Nexus 7
or whatever) that you can easily install things on; that doesn't mean
you want to run your development environment there.

I agree that building native apps will typically involve building in a
clean Ubuntu environment.

> >  * binary packaging format sufficiently similar to existing one that we
> >    could add support to higher-level tools with minimal effort
> >  * strawman design for hooks into system packages, which will be
> >    entirely declarative from the app's point of view
> 
> Is this going to be Ubuntu-specific? If not, you will have some
> trouble with that...

I wasn't planning to put "ubuntu" in the hook paths or anything, but it
is indeed up to the system to declare how it wants apps to integrate
with it.  I don't see a sensible way around that; it is entirely wrong
to put system/app integration code in the installer itself, and any such
approach would result in problems with the system and the installer
getting out of sync further down the line.

dpkg is right to defer all integration code to packages; the hook scheme
I'm considering is mainly inspired by dpkg triggers.  (We may or may not
be able to call dpkg triggers directly; for performance it is vital to
avoid having to actually open the dpkg database.)

> > Footnote: Why not use one of the existing systems out there?
> >
> > We still might, but at present it's not clear that it would make a whole
> > lot of difference.
> 
> If it doesn't, why don't you use them? :P

It's a trade-off between the benefit of using an existing system and the
cost of twisting it around to fit what we actually need it to do.  I'm
not fundamentally opposed to using an existing system - I certainly
understand why people want us to, and that there's a perception that
Canonical unnecessarily does its own thing (you can look up my own
commit record if you like) - but if there's a mismatch between our goals
then it may end up being rather more trouble than it's worth.  One thing
I'm particularly concerned about is that a system that has tried to be
very general may end up reinventing similar scaling and reliability
problems as the packaging system we already have.

(To be clear - hi Slashdot - I'm very fond of the dpkg/apt system and
have put a great deal of effort into it over the last decade or so, both
on my own time and for pay; I think it's excellent and have no desire to
kill it.  But I also recognise that it offers all the rope you need to
hang yourself with, and its failure modes in the face of input from
inexperienced developers are sometimes a little on the abrupt side.)

It's also worth noting that I have had roughly one response for each of
several alternative projects so far; the field is clearly already highly
fragmented.  Not that I think "let's add another standard" is a good
response to that in general, but it does mean that the result of picking
an existing system is that the field is *still* fragmented; not much
realistic difference either for us or for app developers.  So I don't
think this can be a dominating criterion.

> > Many of the existing app packaging systems are more of a reflection of
> > the system they were built for than anything else.  If you look at, say,
> > Android's APK format, it's essentially a zip file with a manifest and
> > some conventions about Dalvik class installation and the like.  Most of
> > the other mobile app formats are similar.  Things like Listaller might
> > be a reasonable fit, but I'm worried about importing things like a full
> > dependency solver into this: much though I love dependency-based
> > packaging systems, they necessarily involve scanning the system
> > packaging database at some point and I would much prefer app packages to
> > be as independent of that as possible, mainly for performance but also
> > to be as bulletproof as possible.
> 
> That is the plan :-) The current Listaller dependency-solving is in a
> bad shape, because it relies on the native package management, and
> only RPM-based distros can deliver all required information for
> Listaller to be able to solve dependencies.

Is this because you're using file dependencies or something?  In most
other respects dpkg is at least as capable as RPM.

> The plan for the next release (which was defered, it should have been
> available in the previous release, but didn't make it), is to really
> simplify dependency solving. This is done by defining "Frameworks",
> which is e.g. KPlatform5, GNOME3, etc. which have to be installed
> using the system package manager (Listaller will fail if no sufficient
> version of them is present). These frameworks are defined through
> small textfiles, which describe the items which have to be present to
> consider a framework as available. Distributors and authors of the
> upstream software will have to write these files.

Right, this is equivalent to the skeletal "Click-Base-System" notion,
and I was also expecting to use the presence of a simple text file to
define that.  (I'm not wedded to the name; I've seen both "framework"
and "profile" used to describe this, and either is probably a better
name.  I'd be fine with adopting an existing standard here.)

> The second aspect is "Modules", shared software which depends on
> Frameworks and which Listaller can try to satisfy using the native
> package manager.
> People should, if possible, only use Frameworks, but for some
> packages, being able to depend on a shared module seems to be
> desirable (e.g. the OggVorbis libraries can easily be satisfied via
> Listaller).
> Frameworks support will definitively be available in the next release,
> Modules might be added later.

This specifically concerns me, and is one reason I'm sceptical about
using an existing system that might have this kind of feature or be
tempted to add it for the sake of being more complete.  It means that as
soon as any app decides to use a module, you suddenly have to open the
system package manager, and deal with a large percentage of the
associated problems (for example, you need to deal with ABI changes in
the additional system packages).

I realise that people are concerned about bloat in apps.  I would say
that we need to deal with this by making the Ubuntu SDK sufficiently
capable to do what people actually need on phones and tablets.  For
later desktop convergence, I do want this system to *work*, but I don't
think it's a priority to make it meet the requirements of everything
people might want to write.  One package might want Vorbis; one might
deliver an Emacs addon; one might rely on a complex and unusual language
framework with lots of loadable modules.  Where do you draw the line?
If you try to make everything available, you are rather likely to end up
reinventing the system package manager and gaining little in the
process.  My preference is to draw the line quite conservatively, at the
SDK which we document on developer.ubuntu.com for app developers, and
not to worry if it doesn't currently meet all the needs we might have in
the future; we can always extend the SDK, or with something like the
frameworks-declared-in-text-files idea you mentioned other people can
declare their own.

> > The proof of concept I wrote also isn't entirely new code.  It's tiny
> > due to using .deb as a container format (minus maintainer scripts, full
> > dependencies, etc.), so I get to save effort by using dpkg to unpack
> > things, which leaves us room to selectively use more of its features in
> > future if we want to.  This is quite a compelling approach to me for
> > various reasons.  I could be persuaded to look at something else, but I
> > don't think there's actually much code to share at the packaging format
> > level anyway, as it's so simple; quite possibly more at the sandboxing
> > level, but that's really a separate layer.
> 
> Hmm... This will make it very difficult for other distributions to use.

I don't think so, really.  dpkg is straightforward to build elsewhere,
so it's no worse than LSB packages, which were RPMs; actually easier
since in this case you don't need to either have a system dpkg database
or fake it by converting the package into some other format first (e.g.
alien).  Yes, LSB packaging wasn't exactly a raging success, but it
failed much more due to lack of interest than due to problems with the
packaging format.  (LSB compatibility has been either available by
default or very easily installable in Ubuntu since the project started.)

I think most ordinary users would have to look up what to do with a
Listaller IPK if they saw one and would still have to go and install
some software first to make use of it, so this seems like a neutral
point to me.

Cheers,

-- 
Colin Watson                                       [cjwatson at ubuntu.com]



More information about the ubuntu-devel mailing list