Persistence in charms

[Clint Byrum]

Excerpts from Brad Crittenden's message of Tue Feb 21 14:11:42 -0800 2012:
> Hi,
> 
> For our work with the buildbot master charm we'd need the ability to save history before the master service is torn down and restore it when it is re-deployed.  I've been told there has been some discussion about direct support in Juju but I've not been able to find the threads.  A pointer to that discussion or a summary as to the plan would be very helpful.
> 

I think we've just had wild, beer/redbull fueled discussions, but nobody has
been willing to go too deep into actual planning because there is so much we
have to do.

> I tried rolling my own store via scp but find there are too many pitfalls (key management, issues with ssh/scp and user identities, etc) and unattractive aspects.  I'm switching gears to provide persistence using boto to an S3-like store.
> 

We *do* need to abstract object storage just like we do with compute for
machines. It should be fairly easy, and those discussions, while also
verbal, did produce some notes that need publishing at some point. The
general idea is that you'd have two commands:

get-object http://some-website/foo

And it would download foo from that url, and cache it in object storage
for the next time somebody requests that url. This makes the system more
robust and independent of remote network failures.

put-object /var/lib/mything/myfile.txt

Would return a url that would be fetchable with get-object. This would
be useful for things like sharing big data blobs across relations, as
you could stick that url into the relationship and the other side can
then fetch it.

The files would be in S3 for EC2 (including whatever you've specified as
your S3 provider for OpenStack) and the webdav service for Orchestra and
the local provider. Because all providers already have to define object
storage for charm storage, this is a pretty low hanging fruit to make
charms more useful and robust.

Unfortunately, we haven't even had time to write this down in a spec. :-P

> When running a non-local (ec2 or openstack) do charms have access to the access-key and secret-key?  Would it be possible, even for lxc deployments, to add support for an external store via the environments.yaml file that the charm can access?  Providing that configuration via the environments file would incur less exposure to secrets than putting them in a config file.
> 

Technically no, they should not have access to those details. In practice,
they're in Zookeeper, and everybody has full read access to Zookeeper,
so you could get them out. However, that is not a guarantee, and will most
certainly be shut down soon as we get more serious about security in juju.

For now, your best bet is probably to wait for subordinates to land (VERY
SOON!) and then you can write a subordinate charm that will send any file
on the filesystem elsewhere. I plan to use this to write a bacula-client
subordinate charm and then use that for backups on a cluster of machines.