Help me understand formula hook idempotence

[Clint Byrum]

Excerpts from Nick Moffitt's message of Sun Aug 21 12:59:24 -0700 2011:
> I am reading the example formulae in lp:ensemble at the moment, and
> noticing a few things that confuse me.  I also see work going on in the
> Principia repository, producing things like
> http://bazaar.launchpad.net/~zulcss/principia/oneiric/nagios/trunk-1/view/head:/hooks/nagios-relation-changed
> which appear to drag in some Python infrastructure in preference to
> shell scripting, so perhaps it's just that the example formulae aren't
> meant to be considered for production use.  
> 
> Either way, two things give me pause.
> 
> First, reading from:
> http://bazaar.launchpad.net/~ensemble/ensemble/trunk/view/174.7.26/examples/mysql/hooks/install
> > echo mysql-server-5.1 mysql-server/root_password password $PASSWORD | debconf-set-selections
> > echo mysql-server-5.1 mysql-server/root_password_again password $PASSWORD | debconf-set-selections
> > DEBIAN_FRONTEND=noninteractive apt-get -y install -qq mysql-server
> 
> How does this avoid races and conflicts?  What if some other formula
> sets the root password?  How do you perform static analysis to prevent
> collisions before a set of changes goes out?  In Puppet (the system I am
> currently most familiar with) the --noop catalog compilation would raise
> an error and prevent application, but I can't figure out how to trigger
> any sort of --dry-run or --simulate options in ensemble.  

Hi Nick! Thanks for taking a look, and great questions...

A formula is basically in charge of everything that /etc/puppet is
in charge of in puppet. Since we're trying to configure a service,
and not a server, it wouldn't make much sense for two things to set the
"root" password for mysql. If a formula needs two admin users, it should
create two.

I do understand conflicting changes and how maddening they can
be. However, formulas are a little more focused on a singular thing
(the service) and not on making a server do many things. If a formula
gets so complicated that it has that much going on inside it, we should
probably make use of tools to do exactly what you're talking about and
avoid conflicting changes.

> 
> What do you do if mysql-server is already installed (perhaps via a
> dependency) but not configured to your desires?  How do you prevent two
> formulae from installing packages that have mutual conflicts/replaces
> settings?  It seems impossible to predict how this hook will behave,
> from my naïve reading.

Hooks that don't assert their intentions are going to cause problems.
An install hook should not say "if mysql is not installed, install
it *AND DO OTHER THINGS*". Idempotent hooks must express it more like
"make sure mysql is installed. make sure mysql has setting x=y. make sure
mysql has user root, and make sure mysql user root has a password of foo".
Its clear to me that many of our formulas don't do that right now, but
that is in large part because we haven't been managing these services
very much. The paradigm of

echo foo/bar string baz | debconf-set-selections
apt-get install foo

is quite problematic. It should actually be

echo foo/bar string baz | debconf-set-selections
apt-get -y install foo
dpkg-reconfigure foo

So that the package is definitely configured in the desired way.

A lot of this is mitigated by the fact that formulas, at the moment,
are not co-installable. That is a discussion that we're having right now,
about how to co-locate two formulas inside the same service. The use case
for this though, is unlikely to conflict.  We're trying to solve things
like instrumentation and logging that are out of band with services. So,
for instance, you may want to colocate munin-node so that all of your
service containers (be they machines or LXC containers inside VMs) have
munin-node available. It really isn't related directly to any of the
services, its more about instrumenting the container and/or the machine.

As of now, you can't really do that.. but its work that is planned for
the near future.

> Second, we also see in the above install hook:
> > sed --in-place=old 's/127\.0\.0\.1/0.0.0.0/' /etc/mysql/my.cnf
> 
> This is what leads me to suspect that this example is not meant for
> production use.  To my eye, the use of sed(1) sticks out as something of
> an anti-pattern: the configuration management world has by and large
> abandoned CFEngine's procedural-edit approaches, choosing instead to
> reliably assert the complete state of system resources.  

Agreed. If you look in the principia formula, it uses augeas to assert
the settings that the rest of the formulas assume are set this way:

augtool <<EOF
defnode mysqld '/files/etc/mysql/my.cnf/target[. = "mysqld"]'
set \$mysqld/server_id $unit_id
set \$mysqld/log_bin /var/log/mysql/mysql-bin.log
set \$mysqld/default_storage_engine InnoDB
set \$mysqld/bind-address 0.0.0.0
set \$mysqld/skip-name-resolve ""
save
EOF

> 
> This allows you to quickly reason about the resulting state in a way
> that gives you confidence in your infrastructure, and ensures that it
> only takes a single run on a production server to achieve correctness.
> Furthermore, `sed -i` operations are order-dependent, which makes for an
> entire category of headaches all on its own!
> 

Agreed. The examples though, demonstrate that ensemble is more about
configuring a more ephemeral thing, a service, rather than something
concrete like a server. I do think for storage use cases, we need to
manage it through its lifecycle, including dynamic formula upgrades that
assert all of their assumptions again. One way to do this would actually
be to write a formula using puppet. 

This is a somewhat controversial idea, but one that I can see working
quite well for those already familiar with puppet's tool set. It would
basically just mean the hooks would install a puppet rule set, and use
facter to save relation data as facts, then at the end of each hook,
run puppet. This allows you to keep using your tools, but also use the
formulas written by other people who maybe aren't so excited about puppet.

> Puppet has support for Augeas, which is a more clever approach to
> CFEngine-style edits: it allows you to assert individual sub-sections of
> a file, and solves the order-dependence problem by raising errors for
> conflicts.  But overall even Augeas is frowned upon.
> 
> The Puppet solution for piecemeal generation of files is called
> "Concat", and it uses a technique somewhat analogous to dotdee but
> without the inotify hooks (though I can't reliably comment on dotdee as
> I haven't used it).
> 
> If you assemble a complete file from configuration management, you can
> reason about the resulting state from a local branch very quickly
> without needing to query running configuration state. For more of the
> reasoning behind abandoning Augeas for general work, see the comment at
> http://www.devco.net/archives/2010/02/19/building_files_from_fragments_with_puppet.php/comment-page-1#comment-5013
> which begins "Augeas combined with Puppet is just in general an
> anti-pattern in my opinion".
> 

This is where we run up against packaging vs. config management. The
mysql package makes certain assumptions about paths, settings, etc. We
take those on as soon as we run 'apt-get install mysql-server'. We
could build my.cnf from scratch, just letting the mysql defaults take
over, but then we need to make sure we take into account the packager's
assumptions. Or we can use augeas, and just actualize our own settings,
rolling forward with the packager's intentions.

The lifecycle of a formula suggests that either approach will meet with
friction whenever mysql is updated. In many ways, the concat approach
is probably more resilient to problems, since packagers are generally
discouraged from doing things against upstream best practice unless it
violates some policy like FHS, which we also wouldn't do in a formula.

Here's where it gets interesting, to me. We only have to solve this once.
Hopefully, we can gather users around a single mysql formula, and they
will find the approaches that work best for most users. If users are
finding that the formula breaks X app or Y app, they can fix it for X app,
or Y app, and if the contribution threshold is kept low (which it should
be given the repository work that is going on right now), we should see
that mysql is fixed for every app with the same needs. Even just having
bug tracking around the formula in a centralized place should allow for
a high degree of collaboration and hopefully prevent the rampant forking
we've seen in most config management repositories.

> I'm still reading through the ensemble code and documentation, so please
> accept my apologies if any of this is a simple failure to RTFM.
> 

No, really you're getting to the heart of problems we're trying to solve
right now in Ensemble. Keep the questions coming!