Opinionated / sensible / recommended default tools for charms

Wed Nov 9 11:31:09 UTC 2011

On Tue, Nov 8, 2011 at 8:37 PM, Adam Gandelman <adamg at canonical.com> wrote:
> On 11/08/2011 08:29 AM, Michael Nelson wrote:
>>
>> On Tue, Nov 1, 2011 at 5:29 PM, Michael Nelson
>> <michael.nelson at canonical.com>  wrote:
>>>
>>> Just for anyone interested, here's part 1 of my juju+puppet experiment
>>> (with a 2min demo):
>>>
>>>
>>> http://micknelson.wordpress.com/2011/11/08/experimenting-with-juju-and-puppet/
>>>
>>> It's mostly pretty obvious - I just use the install/config-changed
>>> hooks to defer to puppet... but it helped me learn much more about
>>> puppet itself and how I can use it together with juju, as well as
>>> start thinking how I could provide an *optional* puppet-master (given
>>> that this will be nearly always private).
>>>
>>> -Michael
>>>
>
> * WARNING: this ended up much longer than I had anticipated.   *

Heh, I'm glad to be able to read someone elses thoughts about this :-)

>
> Michael-
>
> Cool!  I'm glad there are other people scratching their heads about this
> one.  I've played around with a few different ways of doing this (mostly
> exactly like yours) and ran into a couple of concerns.
>
> 1. Stuffing puppet manifests into charms and calling them in hooks via
> 'puppet apply' loses the ability to maintain and share puppet types and
> resources.  Unless you've set a constraint to only depend on puppet
> primitives built into puppet-core, you cannot make use of Puppet's shared
> name space for modules, types and resources(one of the features that makes
> Puppet so powerful)  For example, lets say I've put together a Puppet class
> that incorporates other custom types, providers and modules.  When
> instantiated with a couple of parameters, it propagates a huge chunk of
> system configuration that would otherwise be a nightmare to deploy using
> shell scripts.   Now I want to make use of that class from manifests
> contained within a Juju charm.  There is no way to do that unless I also
> ship the custom module and all of its custom dependencies within that charm
> alongside the manifests that called via 'puppet apply'  If I want to use
> this in multiple juju charms, I need to include the module and its
> dependencies in every Juju charm that uses it.  This quickly becomes a
> nightmare to maintain and impossible to avoid divergence.

Similar to Clint's point, I was expecting/hoping that things like a
puppet apache module would be packaged so that the charm would *make*
it available as part of the install hook. That is, an install hook
something like:

{{{
#!/bin/bash
sudo apt-get -y install puppet

juju-log "Grab any shared puppet modules required for this charm"
puppet apply $PWD/hooks/set_required_puppet_modules.pp

juju-log "Initialising machine state."
puppet apply $PWD/hooks/initial_state.pp
}}}

>
> 2. Calling 'puppet apply' at hook execution only enforces that configuration
> when the relation's state changes. Ie, I've deployed our massive application
> stack on a Friday and gone on vacation for 2 weeks. On Monday  Jr. SysAdmin
> deleted the wrong config file on our web server.  With a puppetmaster in
> place, his mistake is corrected in ~30 minutes (or the next time the agent
> checks into the puppet master).  Without, I'd have to remove and re-add Juju
> relations to resync system configuration with the Juju's view of the world.

I don't think I've got my head around the puppetmaster + juju scenario
yet, but tbh, I was scratching my head about something in between. The
scenario I have is an OSS project for which I want to enable juju
deploys - and I'd like that to be part of the branch so devs can play
with it. But of course there are some aspects of a real deployment
that can't be in the branch itself... so I'm hoping I can use a
conditional in the manifest to do something like:

if $has_puppet_master {
    file { "/srv/whatever.example.com/project/conf/main.cfg":
      source => "puppet://server/modules/whatever/conf/main.cf"
    }
else {
    file { "/srv/whatever.example.com/project/conf/main.cfg":
      source => "file:///wherever/trunk/branch/is/conf/main.cfg"
}

which can probably be summed up as, I don't want to require a puppet
master (so that any dev can spin up and check a production-like
deploy), but I do want to enable spinning up an exact replica of a
production deploy (so that devs with privs/access can test the exact
deploy that the separate IS team would do). Whether the best way to do
that is something similar to above (optional puppet master for configs
etc.) or not, I can't say (as my puppet knowledge is quite limited
atm).

Cheers,
Michael

>
> I'm not sure what the solution is. I've got a few ideas, but they all mostly
> end up reproducing work across multiple domains.  What I initially
> brainstormed:
>
> a. All data gathered from config-get is stuffed custom facts which are
> persistent and updated every time the corresponding hooks are executed (ie,
> $juju-relation-db_user, $juju-relation-db_passwd)  Hooks do the bare minimum
> configuration necessary to get the service up and running (seems to be the
> story of most charms currently)
> b. Units also get a custom fact that describes their service-unit, to be
> used for node classification.
> c. The charm (or, possibly, a colocated puppet-agent charm), authenticates
> the agent with a central puppet master and runs the puppet agent in a
> typical polling configuration.
> d. The puppet master classifies nodes based on service unit, and enforces
> the basic configuration (what is stored in facts) as well as any other
> environment-specific configuration admins may want.
>
> This configuration ends up being maintained over time until the relation
> changes, at which point the facts are updated.  The corresponding puppet
> code on the master may be something as simple as using Augeas to manage
> individual settings in a config file, or some more involved resources that
> deploy large chunks of config that are too cumbersome to express and
> maintain in the juju charm.
>
> However, saving information about the environment on disk (like relation
> data) is a bad idea.  It would be much better if there were some way for
> units to access that data outside of hook execution, so that puppet could
> inspect it client-side dynamically via custom providers (eliminating point a
> above)  It would also be great if Juju could be used as an external node
> classifier (eliminating b above).   Then, we're no longer coupled with
> Puppet and facter and the remaining items (c. and d.) could be expressed in
> Chef-speak just as easily.
>
> I think its important to think about the problem in more general terms.  To
> me, this isn't really about "how can we integrate this with Puppet," but
> "how does configuration management fit into the Juju stack."  To many
> people, there is a gap that needs to be filled before they'll consider Juju.
>  Personally, I'd love to see Juju driving configuration management (chef,
> puppet, whatever) similar to how it drives machine providers (ec2,
> orchestra, whatever) There was some talk about this topic at UDS and I
> expect there will be more as Juju gains a wider audience.
>
> -Adam
>
>
>
> --
> Juju mailing list
> Juju at lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju
>

-- 
Michael Nelson
Canonical Ltd.
+49 176 491 53481 (mob)
michael.nelson at canonical.com
IRC nick: noodles (noodles775 on Freenode)
Ubuntu - Linux for human beings | www.ubuntu.com | www.canonical.com