[ec2-beta] Automated bundling?

[Ivan Wong]

You can place all your secret files in /mnt, bundle will not include that.

-Ivan.

Christopher Smith wrote:
> Having this available in the Ubuntu image would be nice, but nicer I  
> think would be to have the command available on my local computer.   
> Maybe I'm overly worried, but I dislike having to place my secret key  
> on the instance itself(*).  And why have that extra step to have to  
> login to the instance first?
> 
> How about a command that can be run locally which would activate a  
> bundling script on the instance and make use of my secret key stored  
> locally and securely.  Things like the directories to exclude from the  
> bundle and possibly other parameters for bundling could be set on the  
> instance itself, say /etc/ami/bundle.conf and/or /etc/ami/ 
> exclude.list.  Also there would be a /etc/ami/bundle.d where scripts  
> could be placed if anything needs to be run to clean up the instance  
> before bundling.
> 
> ec2-bundle-upload-register  <user at public-dns> <s3-store> <bundle-prefix>
> 
> and given I tend to keep my AMIs in one particular S3 store, that  
> parameter can probably be taken from a preset/default value somewhere.
> 
> I'll have something that does the above soon, but my bash scripting is  
> rough and ready at best.  So sensible stuff, like checking that the  
> ami-tools are installed, you're not overwriting a pre-existing bundle  
> will probably be missing.
> 
> As an aside, In my AMIs I mount S3 and EBS volumes using mount points  
> in the root partition to allow them to be remounted easily from fstab  
> on startup.  Does anyone know if I can keep the mount point but not  
> any of the volumes using the "exclude" option?  Currently, I stop all  
> services that use the storage, unmount the volumes, bundle the image,  
> remount and restart.  However, it would be nice to be able to bundle  
> while the instance keeps working.
> 
> - Chris
> 
> (*) Off-topic/Pet bugbear for AWS.  In order to access S3 from an  
> instance that instance needs to know my secret key which controls my  
> whole AWS account.  Why not a subsidiary id/key for single S3 bucket  
> access only, rather than requiring the account holder to open separate  
> AWS accounts.
> 
> On 31 Jan 2009, at 06:15, Eric Hammond wrote:
> 
>> ...not that I would object to an Ubuntu package which included a  
>> command
>> to do the bundle and upload of a running instance :)   Some other  
>> public
>> AMIs provide this and I think it's a fine idea.
>>
>> The main considerations that pop into my head include:
>>
>> 1. Figure out how the user provides all of the parameters and key  
>> files
>> by default.  If you require them to pass it all in, then it's not much
>> easier than running the two below commands.  There are a number of
>> software packages out there that have different standards for where  
>> and
>> how AWS access identifiers are specified.  If Ubuntu software
>> standardized on one of these it might be useful for future  
>> development.
>>
>> 2. Decide if the bundle/upload command should also register the AMI.
>> The "ec2-register" command is in the EC2 API command line tools which
>> require Java to be installed (not something I'd want on a base AMI  
>> given
>> the size).  However, if you're writing your own software you might as
>> well access the API directly to register the AMI.
>>
>> 3. Figure out what files and directories should be excluded from the
>> bundle by default.  This depends in part on whether the resulting  
>> AMI is
>> intended to be public or private.  For example, public AMIs should not
>> include .ssh/authorized_keys files but private AMIs might want to keep
>> these around.
>>
>> --
>> Eric Hammond
>> ehammond at thinksome.com
>>
>>
>>
>> Eric Hammond wrote:
>>> James:
>>>
>>> When the image includes the EC2 AMI tools, rebundling a running  
>>> instance
>>> comes down to just a couple commands like:
>>>
>>>   time ec2-bundle-vol              \
>>>     -r $arch                       \
>>>     -d /mnt                        \
>>>     -p $prefix                     \
>>>     -u $AWS_USER_ID                \
>>>     -k /mnt/pk-*.pem               \
>>>     -c /mnt/cert-*.pem             \
>>>     -s 10240
>>>
>>>   ec2-upload-bundle                \
>>>       -b $bucket                   \
>>>       -m /mnt/$prefix.manifest.xml \
>>>       -a $AWS_ACCESS_KEY_ID        \
>>>       -s $AWS_SECRET_ACCESS_KEY
>>>
>>>   ec2-register $bucket/$prefix.manifest.xml
>>>
>>> You can find more in the EC2 Getting Started Guide:
>>>
>>>   http://ec2gsg-creating.notlong.com
>>>
>>> The hardest part of this is getting the private key and certificate
>>> files uploaded to the instance.  Note: If you plan to make the image
>>> public, there are other security considerations not suitable for this
>>> short note.
>>>
>>> The community Ubuntu AMIs on http://alestic.com already come with the
>>> EC2 AMI tools installed and it looks like the official Ubuntu beta  
>>> AMIs
>>> should get them soon :)
>>>
>>> If you want to build Ubuntu AMIs from scratch, I publish the
>>> ec2ubuntu-build-ami script linked to from the "Build Script"  
>>> section of
>>> http://alestic.com which many folks are using.  There is also an EC2
>>> plugin being developed for the new Ubuntu vmbuilder package which  
>>> will
>>> do the same thing when it is released.
>>>
>>> That said, I generally recommend to folks that they not bundle  
>>> their own
>>> AMIs, but first try to use the user-data script hook to have a base
>>> public AMI customize itself at boot.
>>>
>>> Application data should never be stored on an AMI, but should be
>>> accessed through EBS, S3, SimpleDB, or the like.
>>>
>>> --
>>> Eric Hammond
>>> ehammond at thinksome.com
>>>
>>>
>>>
>>> James Wyatt wrote:
>>>> Hello,
>>>>
>>>> Just a suggestion for dev efforts. The biggest barrier for a lot  
>>>> of ec2
>>>> users is bundling their own AMIs. Automating this would greatly  
>>>> reduce
>>>> the technical expertise required to effectively use ec2.
>>>>
>>>> Jim Wyatt
>>>>
>>>>
>> -- 
>> Ec2-beta mailing list
>> Ec2-beta at lists.ubuntu.com
>> Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ec2-beta
> 
> 


-- 
Ivan Wong <ivanwyc at gmail.com>
GPG: 1024D/40510AB7: 88BF A832 50D7 30F0 3850  DE17 049D A727 4051 0AB7