TurnKey Linux's take on Ubuntu appliance development: KISS

[Liraz Siri]

.Andreas Heck wrote:
>> * interactions between incompatible appliance configurations including
>>   how to test a combinatorial explosion of possible combinations.
> 
> This is a big misunderstanding. I don't want that the user can install
> different appliance packages at the same time. Of course you can only
> install exactly one package that provides an appliance of any type. I
> see no sane way to combine different appliances without having a human
> doing the integration and testing it.

Ah, thanks for clearing that up, that makes more sense. A few questions:
how do you prevent the user from trying to install more than one
appliance package? You mentioned using a special wrapper to pre-seed
configuration options. What happens if you try to install the package
through APT? Can other packages depend on this hypothetical appliance
package?

>> * how to synchronize appliance development with the development of the
>>   underlying distribution. You are doing the work of a system
>>   administrator, which usually waits until everything is stable to build
>>   production systems out of your distribution.
> 
> You either wait for your target distributions release or you accept that you
> might have to refine some things as the release day comes closer.
> But that's the same as with ordinary packages.

If we wait for the target distribution to release do you mean that these
appliance packages are not really part of Ubuntu but are in a
special/separate repository?

Anyhow, I agree this isn't an insurmountable obstacle, it's just another
thing that adds to the development overhead, like other aspects of
working with packaging infrastructure for this. I guess if you have
enough resources you can work around that.

>> * how to safely update a system-level configuration that may have been
>>   changed and tweaked since the user first installed it.
> 
> This is a big problem indeed and i don't see a perfect solution to that.
> You could warn the user about those changes like normal packages do when
> they upgrade their configuration files. But this might still be better than
> to provide no upgrade solution at all.

Just to make sure we're on the same wavelength here: we think
appliances should behave the same as a regular instance of Ubuntu that
was "handcrafted" by a knowledgable admin. We're just giving users a
better starting point.

That means you can update an appliance at the package level just fine
(apt-get dist-upgrade). You can tweak the system and you don't have to
worry about an upgrade messing around with your configuration. That is
the safe thing to do.

So when we're talking about upgrading an appliance make sure you don't
confuse package level upgrades (which are possible and relatively safe)
with a hypothetical appliance-level update which will automatically
change high-level system configurations.

Since we can't see any safe way to do an appliance-level upgrade in
place (it's a different configuration), what we're working on is making
it easy for users to transfer their data between two different instances
of an appliance.

That way, if a package level update isn't enough you can migrate your
data to a new instance, and after testing that everything works you
repoint (e.g., DNS) to the new instance and shut down the old one.

In the old days where bare metal was the primary target I think it was
more important to be able to make a major upgrade in-place, even though
for mission critical stuff that was *never* a good idea.

Today and increasingly in the future, with virtualization and cloud
deployment, starting and stopping instances is so easy that if you have
a good data migration mechanism there's no real reason to add complexity
and take unnecessary risks.

> I've read that in one of your forum threads you say that you think the best way to
> upgrade is to backup your data and take it over to a new appliance. I totally agree.
> Appliances should more or less be throw away software components and the only
> thing users should really have to care is their data.

Yes BUT we don't want to take appliance-as-a-blackbox and make it too
difficult for users to modify their installations to suit their needs,
because we can't seriously expect to be able to meet 100% of the needs
of our users 100% of the time. Our philosophy is that easy things should
be easy, and harder things should be possible.

> I'm also all for simplicity. But I think there should be an easy and
> reproducible way to build appliances and it would be highly desirable
> if this was based on a default installation of Ubuntu server. This DOESN'T
> HAVE to be packages necessarily. This should also be possible to do with TKLpatch with
> which you are also moving into this direction of reproducible builds or
> some other solution. I just think that then you will reinvent some kind of
> a repository system and if you did that why not implement other package like
> features like reverting this patch and patching the VM to become another kind
> of appliance?

I agree with your point regarding reproducible builds and we are
gradually moving in that direction. TKLPatch is the first step, there
will be others. Note that we started with TKLPatch for practical
reasons. We really wanted to get the community on board and get new
blood into appliance development because there is only so much we can
do. OTOH, the current build infrastructure we're using (inherited from
Ubuntu and Debian) is pretty frigging complex and would be difficult for
the casual developer to reproduce in the current state. We'd like to
work on that too eventually so TurnKey can eat its own tail and build
itself from a set of meta-appliances, but that's further down the road.
Before we do that I think it'll make more sense to improve the
infrastructure and simplify it. Maybe even re-engineer it from scratch,
but that's a big undertaking that doesn't help users directly. Should be
a fun project though - when we get to it.

Meanwhile, TKLPatch is several orders of magnitude simpler and the
casual developer can use it today to tweak stuff to his liking or get
involved in the framework of our project.

> Is there a reason you can't base your appliances on a default Ubuntu image
> and just pull in your infrastructure software that is common to all your
> appliances?

Yes, we're building appliances absolutely from scratch. Our
bootstrapping system is literally the smallest possible system that can
install more packages. Check it out:

http://www.turnkeylinux.org/bootstrap

Also, there are significant technical differences in the installation
methods. Our main target build is an installable Live CD using di-live
(which we developed). Ubuntu server doesn't work like that. It's based
on debian-installer which has a much more complex installation system.
The Ubuntu Desktop also runs live and has an installation system that is
more similar to di-live but needs X to run. That's why we developed di-live.

>> The appliance as a pre-integrated filesystem image is well proven
>> approach followed by TurnKey Linux, SUSE Studio, rPath, JumpBox, and
>> many other vendors you'll find in the VMWare Appliance Marketplace. We
>> already know the pros and cons. It's simple and it works. Today.
> 
> I don't say this doesn't work and it definitely has its place. But I think
> it's possible and desirable to allow users to just say
> "Hey, I want this virtual/physical machine to be an appliance which does THAT"
> and don't force them to download a huge file for every type of appliance
> they want to try out. And since there needs to be some kind of "build system"
> that creates the appliance images anyway, why not make it serve the users
> directly?

Again, the devil is in the details.

Regarding downloading a huge file that's actually more of a problem with
Ubuntu Server. Our Core appliance is about 100MB, and a typical
appliance can add about 50MB to that (more if you need Java frameworks
and the like)

Note that if your hypothetical appliance package pulls in dependencies
over the network using apt-get you are not going to be saving any
bandwidth unless you have a local mirror of the Ubuntu repositories.

Cheers,
Liraz