[ec2-beta] Automated bundling?

[Christopher Smith]

Having this available in the Ubuntu image would be nice, but nicer I  
think would be to have the command available on my local computer.   
Maybe I'm overly worried, but I dislike having to place my secret key  
on the instance itself(*).  And why have that extra step to have to  
login to the instance first?

How about a command that can be run locally which would activate a  
bundling script on the instance and make use of my secret key stored  
locally and securely.  Things like the directories to exclude from the  
bundle and possibly other parameters for bundling could be set on the  
instance itself, say /etc/ami/bundle.conf and/or /etc/ami/ 
exclude.list.  Also there would be a /etc/ami/bundle.d where scripts  
could be placed if anything needs to be run to clean up the instance  
before bundling.

ec2-bundle-upload-register  <user at public-dns> <s3-store> <bundle-prefix>

and given I tend to keep my AMIs in one particular S3 store, that  
parameter can probably be taken from a preset/default value somewhere.

I'll have something that does the above soon, but my bash scripting is  
rough and ready at best.  So sensible stuff, like checking that the  
ami-tools are installed, you're not overwriting a pre-existing bundle  
will probably be missing.

As an aside, In my AMIs I mount S3 and EBS volumes using mount points  
in the root partition to allow them to be remounted easily from fstab  
on startup.  Does anyone know if I can keep the mount point but not  
any of the volumes using the "exclude" option?  Currently, I stop all  
services that use the storage, unmount the volumes, bundle the image,  
remount and restart.  However, it would be nice to be able to bundle  
while the instance keeps working.

- Chris

(*) Off-topic/Pet bugbear for AWS.  In order to access S3 from an  
instance that instance needs to know my secret key which controls my  
whole AWS account.  Why not a subsidiary id/key for single S3 bucket  
access only, rather than requiring the account holder to open separate  
AWS accounts.

On 31 Jan 2009, at 06:15, Eric Hammond wrote:

>
> ...not that I would object to an Ubuntu package which included a  
> command
> to do the bundle and upload of a running instance :)   Some other  
> public
> AMIs provide this and I think it's a fine idea.
>
> The main considerations that pop into my head include:
>
> 1. Figure out how the user provides all of the parameters and key  
> files
> by default.  If you require them to pass it all in, then it's not much
> easier than running the two below commands.  There are a number of
> software packages out there that have different standards for where  
> and
> how AWS access identifiers are specified.  If Ubuntu software
> standardized on one of these it might be useful for future  
> development.
>
> 2. Decide if the bundle/upload command should also register the AMI.
> The "ec2-register" command is in the EC2 API command line tools which
> require Java to be installed (not something I'd want on a base AMI  
> given
> the size).  However, if you're writing your own software you might as
> well access the API directly to register the AMI.
>
> 3. Figure out what files and directories should be excluded from the
> bundle by default.  This depends in part on whether the resulting  
> AMI is
> intended to be public or private.  For example, public AMIs should not
> include .ssh/authorized_keys files but private AMIs might want to keep
> these around.
>
> --
> Eric Hammond
> ehammond at thinksome.com
>
>
>
> Eric Hammond wrote:
>> James:
>>
>> When the image includes the EC2 AMI tools, rebundling a running  
>> instance
>> comes down to just a couple commands like:
>>
>>   time ec2-bundle-vol              \
>>     -r $arch                       \
>>     -d /mnt                        \
>>     -p $prefix                     \
>>     -u $AWS_USER_ID                \
>>     -k /mnt/pk-*.pem               \
>>     -c /mnt/cert-*.pem             \
>>     -s 10240
>>
>>   ec2-upload-bundle                \
>>       -b $bucket                   \
>>       -m /mnt/$prefix.manifest.xml \
>>       -a $AWS_ACCESS_KEY_ID        \
>>       -s $AWS_SECRET_ACCESS_KEY
>>
>>   ec2-register $bucket/$prefix.manifest.xml
>>
>> You can find more in the EC2 Getting Started Guide:
>>
>>   http://ec2gsg-creating.notlong.com
>>
>> The hardest part of this is getting the private key and certificate
>> files uploaded to the instance.  Note: If you plan to make the image
>> public, there are other security considerations not suitable for this
>> short note.
>>
>> The community Ubuntu AMIs on http://alestic.com already come with the
>> EC2 AMI tools installed and it looks like the official Ubuntu beta  
>> AMIs
>> should get them soon :)
>>
>> If you want to build Ubuntu AMIs from scratch, I publish the
>> ec2ubuntu-build-ami script linked to from the "Build Script"  
>> section of
>> http://alestic.com which many folks are using.  There is also an EC2
>> plugin being developed for the new Ubuntu vmbuilder package which  
>> will
>> do the same thing when it is released.
>>
>> That said, I generally recommend to folks that they not bundle  
>> their own
>> AMIs, but first try to use the user-data script hook to have a base
>> public AMI customize itself at boot.
>>
>> Application data should never be stored on an AMI, but should be
>> accessed through EBS, S3, SimpleDB, or the like.
>>
>> --
>> Eric Hammond
>> ehammond at thinksome.com
>>
>>
>>
>> James Wyatt wrote:
>>> Hello,
>>>
>>> Just a suggestion for dev efforts. The biggest barrier for a lot  
>>> of ec2
>>> users is bundling their own AMIs. Automating this would greatly  
>>> reduce
>>> the technical expertise required to effectively use ec2.
>>>
>>> Jim Wyatt
>>>
>>>
>>
>
> -- 
> Ec2-beta mailing list
> Ec2-beta at lists.ubuntu.com
> Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ec2-beta