[ubuntu-cloud] Call for ideas for Ubuntu cloud / [UEC|EC2] images / cloud-init

[Scott Moser]

On Wed, 29 Sep 2010, Simon de boer wrote:

> If it gets more magical....

Well, of course, magical is more fun than simple.  Definitely we want to
make easy and implement once the things that people do over and over
again.

> Suggestion 1:  Having a way to automate the creation of a new base instance.
>  This isn't incredibly terribly time-saving since this only happens every
> once in awhile (although, potentially you need to do it for 4 instance types
> in each zone -- 20ish times currently).

I'm curious why you would rebundle for each instance type, and btw, there
are 10 instance types in my counting.  There are 4 regions at the moment.

> Part of this should be saving  a list of versions to upgrade packages
> (kernels, releases, etc) to -- and using it.  (ie. something similar to the
> Bundler concept from Rails)  Further magic would be a tool to take a
> snapshot of this informati

I'm not sure I'm following.
For building images, there is vmbuilder [1].  For re-bundling our base
images (a starting point that I would recommend) I have a bit of work at
ec2-ubuntu-base [2], but it hasn't seen a lot of time devoted to it.
better tools and documentation for this is definitely a common request.

> Second: To start up "in the cloud" means that after the CPU/OS comes online
> the system has to somehow begin to offer a service.  Unless that service is
> (almost) ridiculously simple it goes beyond one script...
>
> A particularly huge pain point is how to get AWS (or any other) credentials
> _securely_ onto the new instance as it starts up.  These should not / cannot
> be embedded in the base image.  Perhaps the new instance tags is a way
> through this.

Well, I think the IAM (AWS Identity and Access Management) [3] is
definitely amazon acknowledging this need. I personally think that to be
*really* usable, the EC2 portion of this api needs some work.

Creating/Managing credentials would be a client side (prior to launching)
effort, which that we've not spent a lot of effort so far
(uec-run-instances does launch instances, but it is very basic at the
moment [4]).

If you're aware of utilities that do this sort of thing, then those would
be good things for us to package and include in Ubuntu so they would be
easily usable.

For a 'rebundle' app that needed to run inside an instance, I think
generating temporary acls to run snapshot or register or s3 bucket
uploads, and deleting them on termination would be great.

> The next one is monitoring and warnings.  Lets say a new web instance is
> told to start up -- there are many points this could fail:
> * AWS
> * OS startup
> * cloud-init
> * user-script
> * application (ie. apache configuration)
> * load-balancer
> If another system from which you issued the "Start" command was doing some
> monitoring how long should it wait to see the final service come up?  (and a
> approach to monitor such)

This is not made any better by the fact that there is a 4 minute lag
between instance launch and availability of console log.  I really hate
that limitation.

We can definitely add 'phone home' like functionality into cloud-init, so
that it makes a http request to another entity indicating "I'm all ready",
but that doesn't help to know when you should give up on an instance and
just terminate it.  I could see value in the instance posting public keys
so it could be more securely reached without waiting for console output.

the 10.10 version of cloud-init has support for using python's logging [5]
module, and taking a logging configuration from either
/etc/cloud/cloud.cfg or from user-data.  This would allow you to configure
that logging to go to a remote syslog.

> And, more directly, how does cloud-init monitor and scream for help itself.
>  The current approach of having to connect and look through the user.log and
> (if you're lucky) see an error there is tolerable in development...but...

in 10.10 that now all goes to /var/log/cloudinit.log , and as mentioned
above can go elsewhere via syslog or other python logging mechanisms.

One part of cloud-init that I'm pretty happy with, and very interested in
making better, is the ability for you to inject things and change its
behavior from user-data.  In this sense, you could inject a python module
to extend the python logging method, and also specify a python logging
configuration that utilized that.  Then, after '/usr/bin/cloud-init' ended,
subsequent pieces of cloud-cfg and such would use that new logging.

For changing the initial logging of cloud-init, you have to put that data
in the filesystem so it can read it.

> And on the similar topic...there should be a cloud-shutdown as well. (Think
> of a slave database instance going offline -- it will probably need to
> inform application instances elsewhere that this is happening)

Hmm... Is there some portion of normal instance shutdown that you think
would be insufficient for this ?  Ie, on init, you can definitely insert
upstart jobs that would be called on shutdown.

Do you think there would be enough common ground to make a
"cloud-shutdown" useful ?  The bare bones approach I could take would be
to add a call to run-parts of a given directory on shutdown.  Then,
dropping scripts in there would get them called on shutdown.  (there
actually already is something like this -- sysvinit and /etc/rc6.d/).

--
[1] https://launchpad.net/vmbuilder
[2] https://code.launchpad.net/~smoser/+junk/ec2-ubuntu-base
[3] http://aws.amazon.com/iam/
[4] http://bazaar.launchpad.net/~ubuntu-on-ec2/ubuntu-on-ec2/cloud-utils/annotate/head%3A/uec-run-instances
[5] http://docs.python.org/library/logging.html