Juju feedback from the Launchpad Yellow Squad

[Clint Byrum]

Excerpts from Gustavo Niemeyer's message of Wed Feb 15 18:09:38 -0800 2012:
> > Uhh, uhh, wait!!!
> >
> > Stop is not supposed to delete stuff!
> 
> Indeed!
> 
> >> Maybe our collective memory is wrong that juju (the zookeeper box?) has
> >> the old charm cached and does not automatically refresh on an attempt to
> >> redeploy, but that's our impression, and that's what this is about.
> 
> That's an interesting situation indeed. We have to fix this.
> 
> > This has also been a source of frustration for me since my very first
> > charm writing experience.
> >
> > The way it works is that there is a 'revision' file in the charm's root,
> > and that is the only way that juju knows whether or not the charm has
> > changed. The fact that you want to use upgrade-charm is testament to
> > the fact that upgrade-charm was modified to *increment* this revision
> > every time it is run. Thats a nice feature, but only halfway done.
> >
> > IMO, this revision is a blight on the charm format, and should be
> > rethought entirely. To me, we know if any files have changed, and if
> > they have, subsequent deploys should deploy what is in the charm now.
> 
> I'm not sure I understand what you're suggesting. Are you saying that,
> for example, I have a production environment running, and then I hack
> on some files on disk, and then I ask my production environment to
> deploy a service again, and then the files on disk are used rather
> than what is already in the production environment? We really can't
> allow for something like that to happen without explicit
> acknowledgement from the admin. Upgrading a running charm will always
> be an explicit action.
> 

Right, upgrade-charm requires a service name for a reason.. you are
explicitly asking for that service to have the newest version of the
charm.

Deploy creates services.. and whats *VERY* confusing, is that this
command does not in fact deploy the charm on disk, it deploys the latest
one that juju has been told about explicitly.

juju deploy --repository ~/charms local:mysql

That looks like I told it to deploy what is in --repository ~/charms. In
fact, it is what I told it to do. But because I didn't bump revno, juju
is fooled into thinking I haven't changed anything. This seems rather
dense of juju, since I know I expressed my intention quite clearly. This
is definitely the path of great surprise, not least.

There is no warning.. like "using existing charm revision 10", or even
a clue to look for like "sending new charm revision 10". If you forget
to bump revision.. you just get the old one, and it takes inspection
and debugging time to figure that out.

I have been running a single environment now for 3 weeks, trying to
see how the experience is for maintaining the charm store charms and
spinning up new services every couple of days, finding bugs and fixing
them. I wrote the wrapper script precisely because I made this mistake
about 10 times in 2 weeks. Call me lazy (I am!), but having to remember
to bump a file every time I change a local file feels quite absurd.

upgrade-charm is REALLY useful if you want to iterate on an existing
service and just have it re-try the upgrade hook over and over. Or of
course, if you just want the latest charm in an existing service. However,
if you are testing fresh install, its really frustrating to
do this:

juju deploy foo
# notice install_error
juju destroy-service foo
# fix bugs
juju deploy foo
# SAME install error .. major head scratching.. repeat this 5 or 6 times
# and now you start realizing hair is coming out..

> That said, the revision isn't really an oversight, and rather works
> pretty much like you describe already. The revision is important
> metadata for human consumption, though. Please read those two
> sentences:
> 
> "I was using revision ab5766f765 of cs:oneiric/mongodb and I'm now
> using 7efa7fba878a. I've heard revision 15fa87fa8ac is broken."
> 

I do think cs: charms will change this experience quite a bit.  Right now,
we have no cs: experience to go on. For local repos, what is on disk is
what I would expect to get when I do a deploy.. or at the very least,
I'd expect to get a warning that I am not getting what is on disk.

> "I was using revision 10 of cs:oneiric/mongodb and I'm now using 14.
> I've heard revision 12 is broken."
> 
> We may actually introduce a unique digest for charms at some point to
> handle some scenarios, but revision numbers aren't going away.
> 

I don't necessarily want the revno to go away. I want it to make sense
and I want the software to handle them for me. I'd be much happier if
it worked like bzr and when I said 'deploy' it said 'local changes not
deployed' just like bzr warns me when pushing that the stuff I have not
committed yet is not being pushed.

> > I actually just added some code to the juju-jitsu deploy wrapper that
> > just sets the charm's revision to the result of int(time.time()) every
> > time you deploy.  Yes this means every time you deploy you will get
> 
> Ugh.. that sounds nasty. What's the problem with current+1? The plan
> is already to handle local updating of revisions automatically with
> something like that. Whenever you say "upgrade-charm" to a local
> charm, it should bump the revision and send the current content.
> 

current+1 would be fine. I chose time because it was simpler to
implement in code to prove my concept. Thus far it has worked quite
well for charm development. I imagine I have quite a few charms in
S3 now though. :-P I would not necessarily expect this to be the long
term best plan. juju-jitsu is not really a place for long term plans,
its for hacking on stuff and proving concepts.

> > We have thought about adding this into charm-tools as a charm-helper
> > that would inspect a declarative section in metadata.yaml.
> 
> Please don't do that. We may even start blocking charms with random
> content in metadata.yaml at some point to prevent breaking things
> arbitrarily and unintendedly. metadata.yaml is for content that is
> part of the specification only.
> 

Sure, we can keep our little experiments out of metadata.yaml. Though
I'd think at some point some of these intentions will be useful as
metadata for the charm store, and so, it bight be good to gather them
all in one place.

> > So it would look something like
> >
> > apt-packaging:
> >  add-repos: [ 'ppa:charmers/charm-helpers', 'ppa:ubuntu-server-edgers/php' ]
> >  install-packages: ['php5-fpm','nginx','wordpress']
> 
> I don't understand. How's that any better than adding this to the install hook:
> 
>     add-apt-repository ppa:charmers/charm-helpers
>     add-apt-repository ppa:ubuntu-server-edgers/php
>     apt-get install php5-fpm nginx wordpress
> 
> This seems clear, cleaner, and expected.

To understand my reasoning by this, one needs to know how Debian packaging
has progressed from its early days to the state we have now.

Debian packages started out being *just* executables. debian/rules
for building, and debian/{postinst,preinst,prerm,postrm} for running
before/during/after install. debian/control was, and still is, purely
descriptive of packaging information.

This allowed *RAPID* expansion of the package universe.

As this ocurred, the things that people tended to do over and over
started collecting into debhelper. It became painfully obvious that having
declarative lists of files was far more adaptive than having code, so

install -uroot -groot -m 0755 debian/init /etc/init.d/daemon
update-rc.d daemon enable
# and in preinst
...
   invoke-rc.d daemon stop
# and in postinst
   invoke-rc.d daemon start

Which is the right way to install an init.d script, became just this:

dh_installinit

Which would just find debian/init and put it where it needs to go, and
generate the postinst/preinst sections as necessary.

This happened over and over until packages became

dh_dothis
dh_dothat
dh_yetanotherthing
dh_wowthereare20ofthese
... 20 more ...
dh_clean

And then people started making smarter helpers and forgetting some of
the more important dh_ commands.

So then people figured out that most of this was common to each thing,
so CDBS was invented, which turned all that into

include /usr/share/cdbs/classes/debhelper.mk

WHICH nobody liked because it was impossible to read heavily genericised
makefiles. So then they figured out that THIS was all they needed in
debian/rules:


%:
    dh $@


And the *rest* is all in declarative files like 'install' for files
to install, and 'docs' for files that are documentation, and '*.init'
for init scripts. For the odd thing that has to happen differently,
any step can be overridden with:

override_dh_installinit:
    ...do it your way

Why is this better? For the long run, it allows debhelper to adapt to
policy, rather than each and every debian package maintainer having to
keep up to date with policy. Essentially the maintainer trusts that the
helper will get it right. The maintainers end up expressing *intent*
instead of expressing *implementation*.

As an example.. perhaps juju or charm-helper will grow the ability to
cache packages in a clever way so that 'add-unit' is more repeatable.

All the people who have been using apt directly will have to add
MIRROR=`unit-get package-mirror`, Or whatever it is that does that.

However, the users who have used the declarative form will just be
covered by the new version of charm helpers.

> 
> > Then potentially your install hook would just be:
> >
> > # code to install charm-helpers
> > ch_apt_add_repos
> > ch_apt_install_packages
> 
> This is adding more layers of abstraction, more dependencies, more
> code to break, more complexity for the user to understand. What is it
> buying us in exchange?
> 

Same argument, we want to allow charm authors to express their intention
rather than their implementation.

This is less code for the charm author to write, and less code that gets
cargo-culted around and misunderstood. I think we've established that
code reuse is a good thing by now.

Thats why actually eventually I think we'll end up with a more general
command that inspects the charm and does things that are obvious, much
like the 'dh $@' rule above does.