Charm derivation

[Clint Byrum]

Excerpts from Gustavo Niemeyer's message of Mon Jan 16 15:51:26 -0800 2012:
> > It might make sense to fold this work into charm-tools as a django
> > helper. We've found that there are a lot of super-generic things that
> 
> There are dozens and dozens of frameworks out there that have their
> place in the juju world. I'd rather not transform a single project
> into a catch-all place for everybody to throw framework-specific logic
> into, even because hopefully we may even have multiple people pushing
> alternative libraries for the same framework. We don't have to
> centralize that kind of development.

Great point. I do think this would be a good thing to decentralize,
so I retract the suggestion that this particular thing might make sense
in charm-tools.

> 
> > need to be done in multiple charms, like download a file and verify its hash
> 
> Isn't that wget and sha1sum?
> 

The same 15 - 20 lines of shell over and over gave rise to a need to stop
cargo culting this and have a place for snippets of well tested, well
written shell script that can be shared between charms and improved upon
as a whole. Its easier to suggest to people that they use things like:

ch_get_file "http://foo/bar.tgz" "aasdfasdfasdfasd8fasfd8as9d"

Than try to explain how to use wget and sha1sum or sha256sum in a
simple way that works each time and covers the desired use cases. Many
charmers get this wrong, and then their charms suffer for it. There are
other little things, like just being able to get an IP from a hostname
(I know, host x| head -1, except then you also need to know if x is a
hostname or an ip, so you grep, and then...)

These get cargo culted like mad, and sometimes we spot something that
needs fixing in one charm, but then have to go search all the other
charms for the same mistake.  So using a library for this makes sense.

> > or elect a leader of a peer relation.
> 
> That still has a place with juju itself, because it's hard to do
> outside and very easy to get wrong.
> 

Formal leader election is, in fact, quite hard. However, given that we
have some guarantees in the way unit ids are numbered, we can actually
do it very effectively in peer hooks. Because an id is never reused.. we
can always say that we are, or are not, the leader, though pinpointing
who is the leader, when you are not, is actually not reliable because
we can't guarantee the atomicity of relation-list throughout a hook's
execution. So this is not pure leader election of the usual kind, but
it does allow us to say "if I am the leader, do X".

This has proven useful in a few instances, such as with ceph, in picking a
node to initialize the filesystem.  Because this only ever happens once,
we can just say "if I am the leader, do the initialization", and then
even if another node later on thinks it is the leader, it will not do
anything because the filesystem is already initialized. Another example
is when a shared key must be generated and disseminated to all units,
this works to just pick one to be the generator and the others to wait
for a key to arrive. It works because even if the generator dies between
generating and setting, the others will receive a departed event and
have another chance to check if they are the leader and generate the key.

I'm open to suggestions on how to get things into juju faster, but
I figure doing them in charm-tools first has the added benefit of us
charmers actually having tried something before asking for yet another
feature. We've talked about *all* of these things in the past, and agreed
they should be in juju (like a file downloader, and leader election). But
time is limited, and this will suffice for most needs in the short term.

I see this like debhelper for dpkg.. if you tried to put everything
that debhelper does for packagers into dpkg, it would be more robust,
but far too rigid to adapt new features in a timely manner.