updates on production machines
Kent Borg
kentborg at borg.org
Wed Dec 8 15:41:16 UTC 2010
Tom H wrote:
> At the very least, test in a VM before deploying.
Funny you should say that...
A few weeks ago I had my kvm VMs all go bad after a regular update.
None of them would launch. I had to Google around for awhile before I
discovered I needed to add a image file format specification to the XML
files that defined the VMs, and then I needed to Google around more to
find out the exact syntax of the change.
Scary!
The real solution is to have a complete testbed that tries hard to
completely duplicate your production systems--except not dealing with
real transactions, and run all updates through there first. But one
doesn't always get so elaborate, and even when one does, one should
still have a way to roll back.
It got me thinking and I came up with this geeky two-part solution:
1. Use git to keep track of changes in /etc.
2. Before doing an update that worries me, make a snapshot of the
/var/cache/apt/archives directory. If I have the old .deb files I
should be able to reinstall them. I might have to hack around with dpkg
for a few minutes, but I should be able to make it work again.
To do the snapshot I use hardlinks:
# cd /var/cache/apt
# mkdir archives-20101208
# cd archives-20101208
# ln ../archives/*.deb .
The files will only be stored once but there will be two directory
entries point to them. (The day all the directory entries point to a
given file are deleted, the file data itself will finally be deleted.)
One thing to worry about is whether the cache might be cleaned before
you make your snapshot. The apt-get man page says that "apt-get clean"
and "apt-get autoclean" will delete files, as will dselect (not
installed on my computer at the moment). I don't know that the GUIs
don't also mess with the cache. Anything labeled "cache" should not be
relied upon without great thought. To be real thorough one should
probably snapshot .deb files after each backup, when there has been
little chance for cache entries to have been lost.
Doing snapshots with something like LVM is another possibility. Once
btrfs has good userland support, it would be very useful for these things.
-kb
More information about the ubuntu-users
mailing list