brainstorming for UDS-N - Application Developers

[Colin Watson]

On Fri, Oct 01, 2010 at 05:56:27PM +0100, Evan Dandrea wrote:
> I believe that in order to do this properly, we need to massively
> simplify our packaging model.  Anything that makes a package
> non-atomic (hello maintainer scripts) should be thrown out.  Anything
> that adds needless complexity, equally so.  Packaging needs to be the
> least important part of the puzzle, not the most difficult, and most
> certainly not at the core of our own development efforts.

There are few pressing reasons to change the *binary* package format
significantly, and many reasons not to.  The core implementation is
robust yet flexible, knowledge of bits of it is in lots of different
parts of the core system, we use many of its features in non-trivial but
mission-critical ways, and you generally only want to have one package
manager on the system rather than having two of them fighting it out.
Plus I rather suspect that if we tried to reimplement it then the
chances are good that we'd end up in a situation where we had two
package managers neither of which quite met our needs.

The *source* package format, and the tools and processes built around
it, though, are an entirely different matter, and here I largely agree
with you.  And yet: I think that in some ways we have made our lives
unnecessarily difficult by the way in which we present ourselves.  I've
never seen packaging as a particularly important piece of the puzzle: it
is a means to an end.  I have been frustrated for some time about how
the trend in Ubuntu seems to be to present it as something that's
difficult, that needs extensive training, and where there are serious
domain experts, rather than a trivial thing you do to get your code
deployed (and yet, my own attempts at training people have shown fairly
painfully that I shouldn't take up a career in teaching).  Our developer
documentation tends to spend far too much time talking about "some
people like to do it this way instead" rather than providing a single
tutorial and just providing links to reference documentation in case you
need to look up something unusual.  I don't think it needs to be this
way even given our current system.

The trend in source packages over the last several years (thinking of
systems like dh and CDBS) has consistently been towards a more
declarative model, where you merely need to describe the vital
statistics of your package (name, description, top-level dependencies,
etc.) and anything that makes it unusual: if your package has a standard
build system then it's getting increasingly automated.  If you're both
the upstream and the packager then a chunk more of the complexity
disappears.  You still need to write the packaging files by hand, and
know which files to write.  This is one important step, but not the
last.

Quickly allows the packaging files to be generated from templates, so
that developers can use a tool rather than reading a manual.  You still
need to run 'quickly <whatever>' from a command line.  This is another
important step, but not the last.

I can't claim familiarity with all the IDEs for other competing or
historical systems, but I do know that at least some of them have done
their build work by some rough equivalent of writing out a Makefile and
running it.  This should be a hint to us.  We don't need to rewrite our
packaging system - we need to take the current best practice, and extend
GUI development tools to generate that best practice so that application
developers don't need to see any of it, but just download the Ubuntu
SDK, write their code, press a button, and out pops a package.  I
understand that this is more or less the direction Quickly and related
tools are headed, and that's great.  I think effort in this direction
should focus on that, rather than on rewriting our packaging technology;
we would get much more bang for our buck.

There are a number of lower-level improvements that either have helped
or would help, of course.  Triggers made it possible to centralise many
things that were previously distributed across many maintainer scripts,
and got rid of quite a few maintainer scripts altogether.
dpkg-maintscript-helper is a recent innovation that allows simplifying
some very delicate code, and its use would be subject to Quickly-type
automation without much difficulty.  Declarative alternatives and
diversions have been on the list for a while and would help some more.
And, of course, many of these things are not particularly relevant when
working on leaf applications or even on many third-party libraries.

And you're absolutely right that, for the purposes of third-party
application development, a large percentage of this is amenable to
static analysis, and if we can get over the last few percent (which is
mostly licence analysis, susceptible to solution with an SDK that
generated the machine-readable copyright format as one of its tasks),
then I think that we could make the review/acceptance step automatic in
many cases.

> We need to make it our goal that any software developer should be able to
> package their application without difficulty and without causing harm to the
> system.

I don't know if you meant true sandboxing here.  If you just mean
guarding against accidental harm, then the following isn't really
relevant, but I think I saw some other folks mention sandboxing in this
thread so I'll reply to that anyway.

Firstly, sandboxing clearly isn't a requirement for popularity.  As
evidence, I point to Windows and OS X (not that I'm holding up Windows
as a model of technical excellence, but I don't think it can seriously
be suggested that it's short of application developers).  Despite iOS
and Android's ongoing attempts to break into more general-purpose
spaces, I think these are still more apt comparisons than the iPhone or
Android in terms of the kinds of applications we'll be seeing.  Of
course, sandboxing is a good thing if you have it.

So, how could sandboxing be achieved without major binary package format
changes?  I can see several ways to go about it, depending on where you
wanted to put the boundary.  You could define a subset of the format and
enforce it in the client package management tools (see udebs for a
practical example).  You could teach the extras.ubuntu.com archive to
reject any packages that attempt to use more than some limited set of
features.  You could recommend that software developers always use a
particular SDK that never attempts to use more than a particular set of
features, so that any package produced by that SDK is safe (on the
assumption that you don't have a language-level sandbox anyway, so why
bother sandboxing your packaging).  You could automatically generate
AppArmor profiles from the SDK and do static analysis on those at the
archive or client tool level.  Or some combination of the above.

I suspect that the best way to get to there from here is something like
this, in addition to graphical SDK development:

 * Using the early adopters of extras.ubuntu.com as a data source,
   aggressively attack all cases where it's been necessary to write
   maintainer scripts by hand.  Ensure that all of these can be
   generated by either debhelper autoscripts or Quickly templates,
   either of which can be pattern-matched.

 * Write a Lintian mode, somewhat like its existing
   -F/--ftp-master-rejects option, that rejects any package with
   maintainer scripts not consisting entirely of autogenerated
   fragments, or that ship "bad" filenames (outside a particular
   hierarchy, or colliding with existing packages, or whatever), or that
   conflict with packages in Ubuntu itself, etc.  This can be refined
   over time.

 * Modify the ARB processes to fast-track or auto-accept packages that
   pass this Lintian mode (it would take a while to reach the required
   level of comfort, of course).

Although I don't think we'd want to auto-accept things into the core
archive, we could certainly use this kind of thing to improve our core
safety and scalability as well.


In short: accepting mpt's contention that we are behind in attracting
application developers to our system, it behooves us to look for
efficient ways to catch up.  I contend that approaches such as the above
can get us there incrementally, in much less time than it would take to
design and implement a new system from scratch, and without inventing
forward-, backward-, and sideways-compatibility headaches.  The kinds of
problems that Michael Vogt and others face in trying to deal with safe
upgrades from all the crazy things people try to do in packaging should
be addressed by strongly encouraging application packaging via an SDK
that only generates safe packages, and by enforcing safety by analysis
at the extras archive level.  Over time, we can migrate techniques
learned from this into the core system.

-- 
Colin Watson                                       [cjwatson at ubuntu.com]