Juju feedback from the Launchpad Yellow Squad

Gary Poster gary.poster at canonical.com
Thu Feb 16 02:21:26 UTC 2012


On Feb 15, 2012, at 8:27 PM, Clint Byrum wrote:

> Excerpts from Gary Poster's message of Wed Feb 15 13:23:46 -0800 2012:
>> On 02/15/12 15:18, Kapil Thangavelu wrote:
>>> 
>>> Greeting Graham.
>>> 
>>> Excerpts from Graham Binns's message of Wed Feb 15 13:30:51 -0500 2012:
>>>> Hello Juju people (Are we supposed to call you priests? Shamans?
>>>> Anyway...)
>>>> 
>>>> We on the Launchpad Yellow Squad have been working on producing a couple
>>>> of Charms for buildbot (one for the buildmaster, one for buildslaves)
>>>> over the last few weeks. Now that we're ready to submit our Charms to
>>>> you for review, we also wanted to put together some feedback about our
>>>> experience with Juju and with developing Charms. Hopefully it will be of
>>>> some use to you.
>>>> 
>>> 
>>> Awesome, thanks for the feedback, its great to hear about user experiences and
>>> how we can improve juju.
>>> 
>>> 
>>>> Feedback
>>>> ~~~~~~~~
>>>> 
>>>>  * The current Juju Docs are excellent, but they still have significant
>>>>    holes in them. We've spent quite a while trying to get Juju to fit
>>>>    into our collective brain, is a hard but valuable goal. A couple of
>>>>    items missing from the documentation that would be worth including:
>>>>    * Your local terminal needs to be in UTF8 (not by default on
>>>>      my Precise machine for some reason! ANSIX3.4-1968) or else byobu
>>>>      will render the bottom line problematically.
>>>>    * There are several cases in which the best way to learn something
>>>>      is to look at other Charms (e.g. MySQL does a nice job of handling
>>>>      multiple slave services deployed from the same charm but with
>>>>      different configurations). There should be an FAQ or similar
>>>>      pointing Charm developers at good practice example Charms like
>>>>      this.
>>>>  * We spent quite a while doing a lot of work in the config-changed
>>>>    hooks before realising that this wasn't the most efficient way to do
>>>>    things and was prone to breakages (because Juju doesn't keep track of
>>>>    what's changed for you; we implemented our own config caching as a
>>>>    result). Charm authors should be encouraged to use the install hook
>>>>    primarily, with "deploy --config" to set config options.
>>> 
>>> What sort of juju interface would be helpful for changes and why? Afaics its a
>>> fairly trivial thing for a charm todo, but if its a common enough use case, then
>>> juju should accomodate it. Could you clarify the usage that needed this?
>>> 
>>> Ideally, charms should be able to deploy with reasonable defaults even sans
>>> config at deploy time. That doesn't apply to all charms though, but its a good
>>> default.
>> 
>> Our point is not that juju does not support something.  It does 
>> everything we need in this regard (well...see below for a suggestion, 
>> but it's very much a nice-to-have).
>> 
>> Our point is about best practices for charm writing, or at least for 
>> first-cut charm writing.
>> 
>> It can be easier and saner--and, importantly, easier to write automated 
>> tests--if you declare that your charm primarily supports setting things 
>> up via initial configs.  Only certain changes are available 
>> subsequently.  Maybe none are.
>> 
>> This limits the various states you have to handle.  It can let you 
>> provide immediate feedback to a user if the requested state is insane or 
>> broken. It can limit the tests you have to write for a basic feeling of 
>> coverage.
>> 
>> Moreover, we think that charms that follow this pattern will be "good 
>> enough" for the common case.  People who write their first charms maybe 
>> should be encouraged in this direction.  We wish we had gone this way. 
>> It would have been simpler and faster.  We would have identified early 
>> on the config that we wanted to be able to be dynamic, and made 
>> everything else frozen initially.  Really, adding the rleationships is 
>> the primary bit of dynamism that we needed/wanted.
>> 
>> The only thing that I can think of that juju could do to help this would 
>> be to allow a setting of "initial-config-only" for a given config name. 
>>  If the name were not provided initially, juju would refuse to start 
>> the charm; and if the name were set after start up, juju would refuse to 
>> send it, with a helpful error message.
>> 
>>>>  * The one-way communication aspect of relation-set was surprising (so I
>>>>    can relation-set foo=bar but I can never find out later on what I've
>>>>    already told a relation about config options).
>>>>    It would be great if there were a utility to retrieve whatever had
>>>>    been sent to a relation. This would also be a nice way of caching
>>>>    things for Charms :).
>>> 
>>> Could you explain this a bit more, how is it one-way, you can use relation-set,
>>> and retrieve values set with relation-get.
>>> 
>>>  $ relation-get key $JUJU_UNIT_NAME
>>> 
>>> currently unit name is supplied by the default value of the $JUJU_REMOTE_UNIT in
>>> the hook.
>> 
>> Yes, thank you.  I replied to this separately, after our discussion on IRC.
>> 
>>> 
>>> 
>>>>  * Juju should allow service-destroyed hooks: bug 932269
>>> 
>>> the 'stop' hook is the equivalent for a unit: bug 872264
>> 
>> Ah, good to know.  Does juju destroy-service fire the stop hook on all 
>> the units?
>> 
> 
> Uhh, uhh, wait!!!
> 
> Stop is not supposed to delete stuff!
> 
> From here:
> https://juju.ubuntu.com/docs/charm.html#hooks
> stop - Runs when the service unit is stopped. If relations exist, they
> will be broken and the respective hooks called before this hook is called.

LOL, right.  No deletions there.

> 
> Charms don't know what they've touched. Thats going to be a nightmare
> for anything that isn't confined to packages (which do have a nice
> purge method, but even those are very often wrong).

Agreed.

> 
> I think the LXC everywhere idea is actually better than having a
> destroy-relation hook.

The LXC everywhere idea won't work for us, as I mentioned elsewhere.

Perhaps we could just have the ability to really kill units when we destroy services?  So, if the optimization is biting us, we can just at least get new machines?

> 
>>> 
>>>>  * Optimization in ec2 (and presumably Canonistack) of reusing machines
>>>>    is painful, especially without any kind of "destroy service" hook.
>>>>    There's no way to ensure that the previous Charm has cleaned up after
>>>>    itself, which gets particularly problematic when you deploy the same
>>>>    charm to the node later on (because if the old one hasn't cleaned up
>>>>    properly, directories may exist that shouldn't and ports may be bound
>>>>    to that should be free).
>>> 
>>> reuse of a machine is problematic regardless of a stop hook. we'd have to
>>> promote lxc everywhere to get towards this. A container per machine always would
>>> suffice for reuse sans incurring too many networking challenges (forward all).
>> 
> 
> (this is more to Kapil):
> As I've suggested in the past, we need to look into the fact that LXC
> does not actually have to namespace the networking of the container. So
> we can very easily just create containers that just share networking
> with the host. This solves the networking problem, and only creates a
> minor issue where the sshd that runs on the host or in the containers
> will need to be moved to some other port.
> 
>> Note that lxc-everywhere for us would not work, because we need to be 
>> able to run lxc in our charmed services, and lxc does not support 
>> lxc-within-lxc.
>> 
> 
> This *should* work. If it does not, that is a bug in lxc. I have to admit,
> I have not tested this, but it is supposed to work fine.

I thought I heard from Serge that this was not supported, but you're the second person I've heard that from so I'll check with him again.

> 
>>>>  * The upgrade-charm command seems like it needs to work even if service
>>>>    is not active. At the moment you have to have an active instance of
>>>>    the service in order to upgrade the charm, or else you have to
>>>>    destroy-environment and then bootstrap again, which is
>>>>    labour-intensive.
>>> 
>>> juju resolved allows for manually clearing error states and moving a unit to the
>>> started state. its intentional that upgrade only proceeds from a known
>>> good state, charms contain arbitrary logic and can fail in arbitrary ways. if
>>> the service unit is not in a good state, juju provides the tools for a human to
>>> intervene.
>> 
>> This is the use case we are talking about:
>> 
>> [begin developing FOO charm]
>> juju bootstrap
>> juju deploy FOO
>> [discover that our FOO charm is broken]
>> [fix FOO charm]
>> juju destroy-service FOO
>> juju deploy FOO
>> [oops! I forgot to upgrade the charm but I want it to be new when I 
>> deploy the service again! AIUI, the charm is cached, presumably on the 
>> central zookeeper box.  Darn...I have to deploy the service and then 
>> resolve it and then upgrade it, or just destroy-environment]
>> 
>> So, we want upgrade-charm to work when the service is down. 
>> Alternatively and probably even preferable to upgrade-charm working when 
>> a service is not active, "deploy" would always resend the charm, or at 
>> least check for changes and send it conditionally.
>> 
>> Maybe our collective memory is wrong that juju (the zookeeper box?) has 
>> the old charm cached and does not automatically refresh on an attempt to 
>> redeploy, but that's our impression, and that's what this is about.
> 
> This has also been a source of frustration for me since my very first
> charm writing experience.
> 
> The way it works is that there is a 'revision' file in the charm's root,
> and that is the only way that juju knows whether or not the charm has
> changed. The fact that you want to use upgrade-charm is testament to
> the fact that upgrade-charm was modified to *increment* this revision
> every time it is run. Thats a nice feature, but only halfway done.

Ah, ok, thanks for the explanation.

> 
> IMO, this revision is a blight on the charm format, and should be
> rethought entirely. To me, we know if any files have changed, and if
> they have, subsequent deploys should deploy what is in the charm now.
> 
> I actually just added some code to the juju-jitsu deploy wrapper that
> just sets the charm's revision to the result of int(time.time()) every
> time you deploy.  Yes this means every time you deploy you will get
> a new copy of the same old charm pushed to S3, but since I don't know
> what revision was deployed (juju does not reliably expose this to me)
> I have to be a bit coarse until this is resolved in juju itself.
> 
> I think we should probably see how the experience changes with a backend
> charm store available. That may end up making this frustration moot,
> tho I don't know how, since conceivably, charms will be developed locally
> and tested using local: before being pushed into cs:... branches.

Developing locally makes sense to me...but that's for you all to call.

> 
>> 
>>> 
>>>>  * Being able to do a relation-set within a config-changed is important
>>>>    (we understand this is a known issue, but it's worth mentioning
>>>>    anyway).
>>> 
>>> ack.
>>> 
>>>>  * Using environmental variables for identifying the other end of a
>>>>    relation in relation-changed
>>>>    (https://juju.ubuntu.com/docs/charm.html#hook-environment) was
>>>>    surprising, especially given that you can use relation-get to get
>>>>    things like private-address from relation-get.
>>> 
>>> surprising because? relation-get can get data from either end of the relation.
>> 
>> This comment is actually about the fact that some interesting relation 
>> data is exposed via environmental variables, and some is exposed using 
>> relation-get.  Making it uniform would be nice and less surprising. 
>> Specifically, doing it in relation-get would be nice because then you 
>> can document the special names in the help/man text.
>> 
>> Concretely, I can say "relation-get private-address".  Why can't I use 
>> "relation-get private-unit-name" (== $JUJU_UNIT_NAME)?  "relation-get 
>> private-relation-name" (==$JUJU_RELATION)?  "relation-get 
>> private-remote-unit-name" (== $JUJU_REMOTE_UNIT)?
>> 
>> I'm obviously making up and extrapolating a "private-" namespace 
>> convention idea within relation-get, for values that juju provides 
>> itself.  That's for purpose of argument.  Maybe there are better 
>> conventions and names, but hopefully the point is clearer now.
>> 
> 
> I think this is just a product of the fact that before the 'private-*'
> settings, the relation namespace was considered somewhat sacred, and
> so everything that was not specifically relationship data set by charms
> was placed in the environment. That is of course, not the case anymore.
> 
> +1 for adding more private-* fields.
> 
> Charms, however, will have to be careful about relying on this. We may
> need to start having a 'minimum-juju-revision' field in charms as we
> add features like this.

Makes sense.

> 
>>> 
>>>>  * Juju should have a way to share code across Charms, like helpers.
>>>>    bzr does not provide something like svn-external, unfortunately, so
>>>>    this might need to be a build step of some sort. We've got around
>>>>    this by using symlinks during development, but that's no good for
>>>>    production usage.
>>> 
>>> projects like charm-tools and jujuitsu provide for some reusable tools. Beyond
>>> that a charm is free to distribute and install common code via any mechanism,
>>> ubuntu packages, language specific packages, tarballs, etc.
>> 
>> As I mentioned to Mark, packages are problematic when developing. 
>> Tarballs are less heavyweight but still problematic (you have to have a 
>> place to share them).  A lightweight way to say "get this other branch 
>> and make it available before you run install (where I will start to 
>> import code)" is what I want.
>> 
> 
> I think bzr branches in launchpad are a great way to do this until you are
> ready to package everything up.
> 
> They're not so great for users to have repeatable, robust deployments though.
> So while you're developing, this works fine:
> 
> if [ "`config-get hooksource`" = "dev" ] ; then
>  bzr co --lightweight lp:~myteam/+junk/foo /usr/local/lib/python2.7/foo
> else
>  add-apt-repository -y ppa:myteam/foo-helpers
>  apt-get update
>  apt-get install -y python-foo
> fi

Hm, ok.

> 
> Perhaps there is some room for charm helpers to make this easy.
> 
>> Perhaps the following would be a framework, or at least a straw man, for 
>> what I think ought to be there.
>> 
>>  * Juju lets charms simply and easily declare additional deb 
>> repositories (e.g. PPAs) and packages that should be installed before 
>> the charm runs.  (Extra points if the mechanism allows charms to let 
>> these values be added to or overridden by deploy-time configuration.)
>> 
> 
> We have thought about adding this into charm-tools as a charm-helper
> that would inspect a declarative section in metadata.yaml.
> 
> So it would look something like
> 
> apt-packaging:
>  add-repos: [ 'ppa:charmers/charm-helpers', 'ppa:ubuntu-server-edgers/php' ]
>  install-packages: ['php5-fpm','nginx','wordpress']
> 
> Then potentially your install hook would just be:
> 
> # code to install charm-helpers
> ch_apt_add_repos
> ch_apt_install_packages
> 
> Actually even better, I think, would be to learn from debhelper 7,
> and just have a single command in install:
> 
> #!/bin/sh
> exec charm-helper install 
> 
> That does all of the usual tasks based on stuff in the metadata and
> other declarative files.

Cool.

Perhaps the most important aspect of this is to provide a road for charm authors to travel down, however you want to pave it.

Thanks for the replies

Gary


More information about the Juju mailing list