TurnKey Linux's take on Ubuntu appliance development: KISS

[Alon Swartz]

Soren Hansen wrote:
> On Mon, Dec 07, 2009 at 11:12:04AM +0200, Alon Swartz wrote:
>>> Could you elaborate on this a little bit? Lots of different ideas
>>> have been tossed back and forth on various mailing lists, IRC and
>>> other fora, so I'm not completely sure what you mean Debian packages
>>> were not designed for.
>> What this means is that Debian packages were never designed for
>> system-level integration. They're meant to be building blocks for a
>> system administrator who then glues them together to integrate a
>> production solution.
> 
> I disagree. There's nothing in the deb file format, nor the Debian
> Policy that prevents this. It simply defines a very strict set of rules
> for how it can be accomplished. I completely acknowledge, that there's a
> /tradition/ in Debian for treating packages as platform building blocks
> that the administrator is supposed to integrate with each other to form
> a complete solution of some sort. I would love nothing more than to see
> this change in Ubuntu. We've made some progress on the mail server
> front, but I see no reason not to extend this much further.
> 
>> There's no elegant way to do that with packages / meta-packages
>> precisely because they're not designed for this sort of thing. For
>> example, according to the Debian Policy Manual a package is not
>> allowed to edit configuration files of other packages.
> 
> Indeed. That is exactly why we would add customisations to the packages
> themselves, or extend packages to offer an interface for other packages
> to change their configuration (like, e.g. postfix does it with
> postconf). 
> 
>> We love packages, but an appliance is a higher-level concept and it's
>> important to understand that.
> 
> I realise that appliances are a higher-level concept. I just don't
> (yet?) see why that necessarily means that appliance-related
> customisations cannot live in the packages instead of being applied
> out-of-band.
> 
>> For example, let's compare the LAMP stack with the MySQL appliance.
> [...]
>> LAMP stack and the MySQL appliance have very similar components, but
>> different usage scenarios and hence - different configurations.
> 
> I don't see any reason why we couldn't ship both options in the mysql
> package. We could default to the current default and add a low priority
> debconf question asking which type you prefer and preseed that for the
> appliances. That way, all logic pertaining to mysql is contained in the
> mysql source package, but exposed to other entities for tweaking.

There are 2 distinct issues that have been referred to as one, so I
would like to take a step back and discuss them:

1. Extending packages to support different needs

I am all for this. Extending packages to support different use-cases is
a great idea, not a new one. I would love to see more innovation in this
area, appliances aside.

As Liraz mentioned in one of his posts, we initially started down this
path but quickly got bogged down with with package development and
maintenance, instead of focusing on building useful appliances.

Forking packages also created the burden to re-apply our patches
when an update was released upstream, so appliances could upgrade. In
contrast, changing a packages configuration post installation allows for
faster integration, and the ability to upgrade directly from upstream.

This might be due to a lack of resources, as well as being downstream,
but even if you have the resources to spend, is it the best use of them?

2. Using packages to essentially /build/ appliances

As I mentioned before, Debian packages were never designed for
system-level integration, and you seem to agree (sort of) by your
proposal to "add customizations to packages, or extend packages to offer
an interface...".

Its not only different use-case configurations that need to be
supported, its the glue that brings them together into a cohesive,
working solution out of the box.

Forgive me if this sounds cliche, but it reminds me of "when you have a
hammer, everything looks like a nail". Is this a solution looking for a
problem?

Undoubtedly, it would work technically, we went this route in the past,
but just because something works doesn't mean it is the best solution.

> Anything you have to patch out-of-band in an Ubuntu package for it to be
> useful, I would consider a bug. Within reason, of course.

In essence I agree, but this is not the case. The patches are not
intended to make the package /useful/, they are intended to make the
package /useful/ for the specific use-case the appliance is designed to
meet.

>> Whenever a package is not supported in Ubuntu or worse, doesn't even
>> exist as a Debian package, that makes it much more difficult to create
>> and maintain appliances.
> 
> Definitely. For anything to be even considered an "official" Ubuntu
> appliance, it would have to consist only of software available as
> packages in the repository. It is the only way we can reasonably support
> anything.

Completely understood and agreed.

>> We don't have the resources to package all of this software ourselves
>> in our own repositories, so we've done that only for very popular
>> software packages such as Joomla.
> 
> I would love to see a Joomla package in Ubuntu. In fact, I would love to
> see any and all the software you've appliancified as packages in Ubuntu.
> I think appliances should be a convenient way to deploy certain software
> stacks, not the /only/ way to deploy it.

Agreed. What would be the best way to push this forward?

Any custom package that isn't taken straight from Ubuntu's repositories
can be found in our mini-repository [1].

The source code to all custom components can be found in our code
repository [2]. Some components are also hosted on github [3].

[1] http://archive.turnkeylinux.org
[2] http://code.turnkeylinux.org
[3] http://github.com/turnkeylinux

> Have you usually found the dependencies of your software to be met, or
> do you usually have to install extra Perl, PHP, or Python modules to
> meet the needs of the software you're turning into an appliance?

At the package level, yes - dependencies are satisfied as expected. If
not then its considered a bug and we will report it.

There are certain cases depending on the appliance (and the use case)
where we install extra packages to either add more functionality (mostly
taken from recommends or suggests), or packages that are required to
fulfill the requirement of the appliance (e.g., python-pastedeploy to
support proxying through apache to loggerhead in the revision-control
appliance).