TurnKey Linux's take on Ubuntu appliance development: KISS

[Liraz Siri]

Andreas Heck wrote:
> There is no question that packages that create appliances and therefore
> do integration of other packages are a very similar but still a
> different concept than the "normal" software packages Debian uses for
> ages. This means that if you use packages for packaging appliances you
> cannot apply the same standards as you apply to normal packages (like
> the Debian policy manual e.g.). This is also something I'm not perfectly
> happy with, of course.

Watch out. In our experience breaking packaging standards can have
consequences that may not be immediately obvious but will come back
later to bite you. You should be very cautious about that.

> Nevertheless I still think it is a good idea to use packages for this
> job. The only problem with packages I found so far apart from doing
> things you wouldn't do in other packages is that you most often need to
> seed your debconf database before you can install the metapackage that
> creates the appliance by pulling in the right dependencies and
> integrating them with the help of its maintainer scripts.
> 
> For the moment I solved this problem by putting a seed file into the
> same directory of the repository as the metapackage and using a very
> small wrapper script around apt-get to do the actual installation which
> takes care of downloading and applying the seed file before the actual
> installation takes place. Of course you might be more invasive and put
> the file into the package and maybe even patch dpkg to recognize it or
> use another wrapper the pulls the seed from the package.

Yes, I read your spec here:

https://wiki.ubuntu.com/VirtualAppliancesAsPackages

You can preseed debconf configuration variables in a tklpatch as well.
Unfortunately usually the amount of configurability provided via debconf
is fairly limited. Once you start building appliances to satisfy
real-world usage scenarios it becomes clear that most configurations
needs a more flexible, open-ended approach. TKLPatch embodies that.

> If you want to do stuff like that you come into a situation where you
> reinvent some kind of a package system very quickly. This is because you
> need to have some kind of repository where you can get a list of
> available appliances from and you also need to be able to pull in the
> integration code that creates the appliance on top of an existing
> default installation of Ubuntu. And maybe you even want to turn your
> appliance into a "normal" server again or turn it into a completely
> different type of appliance.

This sounds fantastic, but the devil is in the details. There are
substantial problems with the appliance as a modular package approach
that may not be obvious at first glance.

We went through a similar stage in our thinking before we launched
TurnKey Linux, but after working on a few prototype appliances it became
clear that this approach added a large amount of unessential complexity
to the development process. You have to think about:

* interaction between an infinite set of possible starting points and
  your "appliance"
* interactions between incompatible appliance configurations including
  how to test a combinatorial explosion of possible combinations.
* how to synchronize appliance development with the development of the
  underlying distribution. You are doing the work of a system
  administrator, which usually waits until everything is stable to build
  production systems out of your distribution.
* how to safely update a system-level configuration that may have been
  changed and tweaked since the user first installed it.
* what the development cycle for this will look like.

Our conclusion was that the appliance as a package approach
unnecessarily complicates all aspects of development. Complicated things
are hard to understand. They're hard to develop, hard to maintain, hard
to get right. You pay for complexity with reduced productivity, reduced
reliability and reduced security.

So instead we decided to do the simplest thing that could possible work.
In other words KISS - Keep It Simple Stupid.

At TurnKey each release batch starts with Core, a single unchanging base
that includes all the standard features. Our development cycle is
heavily based on virtualization. We snapshot, apply changes to Core,
test the result, find a bug, refine our change-set, rollback, rinse,
repeat.

A TurnKey appliance isn't opaque. We don't own your system-level
configurations, we just give your a better starting point.

> This makes other interesting use cases possible, too. So you could just
> put some appliances on the Ubuntu server CD and allow the user to select
> one if he wishes this server to be an appliance in fashion very similar
> to tasksel. It would also allow to take a virtual server as you can get
> it for about 10€ per month and turn it into an appliance. We could even
> create a nice looking GUI program where you select an appliance from a
> list, provide your credentials, and the program than transforms your
> cheap virtual your server into the appliance you wish.

The appliance as a pre-integrated filesystem image is well proven
approach followed by TurnKey Linux, SUSE Studio, rPath, JumpBox, and
many other vendors you'll find in the VMWare Appliance Marketplace. We
already know the pros and cons. It's simple and it works. Today.

Doing it this way cleanly decouples your appliance from any messy
OS-level / inter-appliance interactions and leverages all of the
innovation happening in the virtualization industry.

In fact deploying self-contained units of functionality that only depend
on the underlying (virtual) hardware is sort of the main reason for the
industry trend towards appliances in the first place:

http://en.wikipedia.org/wiki/Software_appliance

The use cases you mention are interesting, but they're an example of
premature optimization. We could hypothetically save a few DVDs by
making a one-size-fits all DVD that assembles appliances on-the-fly from
a pool of shared components. I mean it sure sounds cool, but is that
really the problem we need to be solving?

I think you should first exhaust what's possible within the current
proven paradigm and invent a new one (e.g., virtual appliance as a
special Debian package) as a last resort.

IMHO, we haven't reached that point yet. For example, all of the use
cases you mentioned could also be satisfied within the current paradigm:

* We're already working with cloud providers such as Amazon EC2 and a
  range of VPS providers to deploy TurnKey appliances on-demand. We're
  also talking to many other providers both large and small and hope to
  be able to support as many of these services as possible. That's a
  high priority for us.

  Note that emerging standards such as OVF make it easy to support many
  virtualization platforms with a single image format. Currently support
  for OVF is still limited, but that's where the industry is headed and
  you can imagine in the not so distant future a scenario where you have
  databases of online OVF repositories that allows your virtualization
  platform of choice to support point and click installs of a wide and
  diverse range of appliances from many vendors.

* Today, some of VMWare products already allow you to install an
  appliance on the fly, heck even STREAM an appliance on the fly
  straight from a repository of registered appliances. You could
  implement an open source version that.

  For example, one VPS provider repackaged TurnKey's appliances as
  DTC-Xen modules which are stored in a Debian package repository.

* You can write a GUI that downloads a conventional appliance, and
  applies arbitrary changes to the configuration (e.g., using TKLPatch
  as a back-end).

* There's interest in the community in creating a master
  "meta-appliance" that will include a hypervisor and will be bundled
  with a collection of TurnKey (and other appliances).

  For example, here's one discussion about that on our forums:

  http://bit.ly/63J1jw

> Another point that makes packages seem very interesting to me is
> updating. If you have an appliance and someone makes a bugfix to the
> integration code you could use TKLpatch to apply it but as far as I
> understand it you would still have to do that manually. With Debian
> packages you could this automatically and with the normal update
> procedures although you could get into troubles with conflicting
> user-changed configurations with both approaches.

We've had a couple of related discussions in our community you might
want to check out:

http://bit.ly/7uWAY8

http://bit.ly/8qwL8U

Cheers,
Liraz Siri
Cell: +972-54-2013512
Twitter: http://twitter.com/lirazsiri
Google Talk: liraz at turnkeylinux.org (Jabber/XMPP IM)