[ec2-beta] ec2 & upgrade testing

[Michael Vogt]

On Sun, Jan 18, 2009 at 07:04:05PM -0800, Eric Hammond wrote:
> Michael:
Hi,

> Here are the ideas I was thinking about that might help you get past the  
> 10G limit on the root partition:
>
> (1) Run a base image.  On the running instance, copy some of the bigger  
> directories to /mnt and then mount them back over top of the original  
> location.  Install all the packages you need, then test the upgrade  
> process.  E.g.,
[..]

Thanks, that is indeed a nice and simple solution. I think I will do
the initial implementation based on that and then move to the option
(3). 

[..]
> (3) There has been some progress in the exploration of storing an entire  
> root partition on an EBS volume of any size and using pivotroot at  
> startup.  Here are some threads which talk about this:
[..]

I like the ability to use snapshots and having different EBS volumes
with pre-build image to test e.g. a ubuntu and a kubuntu upgrade. I
think I play with (1) to get more experience with ec2 and then move to
(3). 

Thanks a lot for your reply and your help :)

Thanks,
 Michael


> Michael Vogt wrote:
>> On Fri, Jan 16, 2009 at 11:13:13PM -0800, Eric Hammond wrote:
>>> Michael:
>> Hi Eric,
>>  
>>> There are a few ways to get more than 10 GB effectively or actually on a
>>> root file system on a running EC2 instance.
>>>
>>> Depending on your needs and the level of complexity you're willing to
>>> deal with, the data can even be persistent and/or can be used as an
>>> image to start multiple instances (though not exactly an AMI).
>>>
>>> Can you share more about what goals you are trying to accomplish and the
>>> processes you use?
>>
>> Currently I use the vmbuilder to create the AMI and have python code
>> that runs a instance based on this, installs additional packages into
>> it (ubuntu-desktop for example). Then it copies the current release
>> upgrader code into the instance and upgrades to the next version of
>> ubuntu to see if there are any issues with the package upgrades or the
>> upgrader code.
>>
>> What I would like do be able to do is have images/AMIs for the various
>> ubuntu flavours (ubuntu,kubuntu,..) available so that the upgrade test
>> does not have to construct them again. I would also like to run
>> upgrade tests with a lot of packages (e.g. most of main or even most of
>> universe). That can easily go much beyond the 10G limit. Having a
>> snapshoting ability would also be nice but its not that important
>> right now. 
>>
>> I hope the above gives a bit of background, any hints/links to more
>> information how to get around the limit are much appreciated (I'm
>> still pretty new to ec2).
>>
>> Thanks,
>>  Michael
>>
>>  
>>> Robbie Williamson wrote:
>>>> Great work, Michael!  I'm copying the ec2-beta list, as people on it will be
>>>> interested and can also help you, if needed. ;)
>>>>
>>>> -Robbie
>>>>
>>>> On 01/16/2009 08:56 AM, Michael Vogt wrote:
>>>>> Hi,
>>>>>
>>>>> I worked on the ec2 upgrade testing stuff today and yesterday and I
>>>>> have a working prototype now in the lp:~mvo/update-manager/ec2
>>>>> branch. It can test upgrades in a ec2 instance in a similar fashion as
>>>>> the current kvm backend. I created the base images with vmbuilder
>>>>> (from my lp:~mvo/vmbuilder/mvo branch that contains some fixes against
>>>>> trunk/).
>>>>>
>>>>> I did a successful automatic/unattended-upgrade for the server and the
>>>>> ubuntu-desktop profile. But my code is still prototypish and needs
>>>>> some cleanup love.
>>>>>
>>>>> One pretty anoying limiation currently seems to be that the size of a
>>>>> image (a AMI) seems to be limited to 10Gb [1]. That is not enough for a
>>>>> full blown upgrade test that includes most packages in main (it is
>>>>> sufficient for our various default installs).
>>>>>
>>>>> Its a good addition to the kvm based backend but its still useful to
>>>>> keep the kvm one around. Its nice to be able to boot a kvm upgraded
>>>>> image and login to test stuff manually (or verify upgrade bugs).
>>>>>
>>>>> There is still a lot of room for improvements in the ec2 backend. I
>>>>> just use a single base AMI right now for everything, this should be
>>>>> changed so that each base image becomes its own ami. I also don't make
>>>>> any use of the ec2-volumes yet (but I'm not quite sure yet if they are
>>>>> useful for my use-case). I'm still learning how everything fits
>>>>> together :) 
>>>>>
>>>>> Cheers,
>>>>>  Michael
>>>>>
>>>>>
>>>>>
>>>>> [1] 
>>>>> http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1145 
>>>>> (question: "When I try to bundle my AMI, it fails. Why?")
>>>>
>>