[ec2-beta] ec2 & upgrade testing

[Eric Hammond]

Michael:

Here are the ideas I was thinking about that might help you get past the 
10G limit on the root partition:

(1) Run a base image.  On the running instance, copy some of the bigger 
directories to /mnt and then mount them back over top of the original 
location.  Install all the packages you need, then test the upgrade 
process.  E.g.,

   for i in /usr /var/cache /var/lib; do
     sudo rsync -PaR $i /mnt/
     sudo mount --bind /mnt$i $i
   done
   apt-get install ...

Pros/cons:

   + Easy
   - Process must be repeated every time you want to test upgrade
   - Can't relocate directories which have open files (reboot?)


(2) Same as (1) but instead of using /mnt (ephemeral storage), the 
directories could be copied to an EBS volume (persistent storage) and 
clean installs performed.  Take a snapshot of the EBS volume and bundle 
the rest of the root partition as a new AMI.  Figure out a way to start 
new instances with the AMI + new EBS volume created from the snapshot.

Pros/cons:

   + Faster to start up fresh systems (if it's even possible)
   - More difficult to set up
   - Must coordinate AMI + snapshot to be consistent
   - Not sure if the AMI portion will contain tools necessary to mount
     EBS volumes so this approach may not be feasible.


(3) There has been some progress in the exploration of storing an entire 
root partition on an EBS volume of any size and using pivotroot at 
startup.  Here are some threads which talk about this:

http://developer.amazonwebservices.com/connect/thread.jspa?threadID=24091
http://developer.amazonwebservices.com/connect/thread.jspa?threadID=24440

Pros/cons:

   + Everything on a single mount point
   + EBS snapshot captures entire consistent install
   + Faster to start up fresh systems
   - Setup requires the use of advanced methods


Let me know if you'd like assistance pursuing one of these or other 
alternatives.

--
Eric Hammond
ehammond at thinksome.com



Michael Vogt wrote:
> On Fri, Jan 16, 2009 at 11:13:13PM -0800, Eric Hammond wrote:
>> Michael:
> Hi Eric,
>  
>> There are a few ways to get more than 10 GB effectively or actually on a
>> root file system on a running EC2 instance.
>>
>> Depending on your needs and the level of complexity you're willing to
>> deal with, the data can even be persistent and/or can be used as an
>> image to start multiple instances (though not exactly an AMI).
>>
>> Can you share more about what goals you are trying to accomplish and the
>> processes you use?
> 
> Currently I use the vmbuilder to create the AMI and have python code
> that runs a instance based on this, installs additional packages into
> it (ubuntu-desktop for example). Then it copies the current release
> upgrader code into the instance and upgrades to the next version of
> ubuntu to see if there are any issues with the package upgrades or the
> upgrader code.
> 
> What I would like do be able to do is have images/AMIs for the various
> ubuntu flavours (ubuntu,kubuntu,..) available so that the upgrade test
> does not have to construct them again. I would also like to run
> upgrade tests with a lot of packages (e.g. most of main or even most of
> universe). That can easily go much beyond the 10G limit. Having a
> snapshoting ability would also be nice but its not that important
> right now. 
> 
> I hope the above gives a bit of background, any hints/links to more
> information how to get around the limit are much appreciated (I'm
> still pretty new to ec2).
> 
> Thanks,
>  Michael
> 
>  
>> Robbie Williamson wrote:
>>> Great work, Michael!  I'm copying the ec2-beta list, as people on it will be
>>> interested and can also help you, if needed. ;)
>>>
>>> -Robbie
>>>
>>> On 01/16/2009 08:56 AM, Michael Vogt wrote:
>>>> Hi,
>>>>
>>>> I worked on the ec2 upgrade testing stuff today and yesterday and I
>>>> have a working prototype now in the lp:~mvo/update-manager/ec2
>>>> branch. It can test upgrades in a ec2 instance in a similar fashion as
>>>> the current kvm backend. I created the base images with vmbuilder
>>>> (from my lp:~mvo/vmbuilder/mvo branch that contains some fixes against
>>>> trunk/).
>>>>
>>>> I did a successful automatic/unattended-upgrade for the server and the
>>>> ubuntu-desktop profile. But my code is still prototypish and needs
>>>> some cleanup love.
>>>>
>>>> One pretty anoying limiation currently seems to be that the size of a
>>>> image (a AMI) seems to be limited to 10Gb [1]. That is not enough for a
>>>> full blown upgrade test that includes most packages in main (it is
>>>> sufficient for our various default installs).
>>>>
>>>> Its a good addition to the kvm based backend but its still useful to
>>>> keep the kvm one around. Its nice to be able to boot a kvm upgraded
>>>> image and login to test stuff manually (or verify upgrade bugs).
>>>>
>>>> There is still a lot of room for improvements in the ec2 backend. I
>>>> just use a single base AMI right now for everything, this should be
>>>> changed so that each base image becomes its own ami. I also don't make
>>>> any use of the ec2-volumes yet (but I'm not quite sure yet if they are
>>>> useful for my use-case). I'm still learning how everything fits
>>>> together :) 
>>>>
>>>> Cheers,
>>>>  Michael
>>>>
>>>>
>>>>
>>>> [1] http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1145 
>>>> (question: "When I try to bundle my AMI, it fails. Why?")
>>>
>