updates on production machines

[Kent Borg]

Tom H wrote:
> At the very least, test in a VM before deploying.

Funny you should say that...

A few weeks ago I had my kvm VMs all go bad after a regular update.  
None of them would launch.  I had to Google around for awhile before I 
discovered I needed to add a image file format specification to the XML 
files that defined the VMs, and then I needed to Google around more to 
find out the exact syntax of the change.

Scary!


The real solution is to have a complete testbed that tries hard to 
completely duplicate your production systems--except not dealing with 
real transactions, and run all updates through there first.  But one 
doesn't always get so elaborate, and even when one does, one should 
still have a way to roll back.

It got me thinking and I came up with this geeky two-part solution:

 1. Use git to keep track of changes in /etc.

 2. Before doing an update that worries me, make a snapshot of the 
/var/cache/apt/archives directory.  If I have the old .deb files I 
should be able to reinstall them.  I might have to hack around with dpkg 
for a few minutes, but I should be able to make it work again.

To do the snapshot I use hardlinks:

  # cd /var/cache/apt
  # mkdir archives-20101208
  # cd archives-20101208
  # ln ../archives/*.deb .

The files will only be stored once but there will be two directory 
entries point to them.  (The day all the directory entries point to a 
given file are deleted, the file data itself will finally be deleted.)

One thing to worry about is whether the cache might be cleaned before 
you make your snapshot.  The apt-get man page says that "apt-get clean" 
and "apt-get autoclean" will delete files, as will dselect (not 
installed on my computer at the moment).  I don't know that the GUIs 
don't also mess with the cache.  Anything labeled "cache" should not be 
relied upon without great thought.  To be real thorough one should 
probably snapshot .deb files after each backup, when there has been 
little chance for cache entries to have been lost.

Doing snapshots with something like LVM is another possibility.  Once 
btrfs has good userland support, it would be very useful for these things.


-kb