SATA (hot) swapping for backup
ian at syndicated-productions.com
Thu Sep 21 15:56:44 UTC 2006
Hey everyone -- I just stumbled on this thread and thought it might be
helpful to add some of my own experience.
Daniel Pittman <daniel at rimspace.net> writes:
>David Abrahams <dave at boost-consulting.com> writes:
>> Daniel Pittman <daniel at rimspace.net> writes:
>>> David Abrahams <dave at boost-consulting.com> writes:
>>>> Daniel Pittman <daniel at rimspace.net> writes:
>>>>> David Abrahams <dave at boost-consulting.com> writes:
>>>>>> 1. If I boot the server with no drive in the bay or with the drive
>>>>>> power off, it is never detected, even after powering it on. Is
>>>>>> there a way to get that to work?
Yes there is. The issue here is that the scsi bus does not automatically
re-scan when something is plugged in. You can do this by manipulating
/sys/... or using the 'rescan_scsi_bus.sh' script (search the web for it) or
the 'scsiadd' tool (http://llg.cubic.org/tools/). The reason it doesn't
work automatically is because of the lack of kernel hotplug support in
libata and sata drivers prior to 2.6.18.
>>>>>> 2. I'm not 100% sure that unmounting the drive, powering it off,
>>>>>> removing it, and putting a new disk in its place is legit. Can
>>>>>> anyone confirm? My motherboard *does* claim to support SATA
>>>>>> hotswap, but I'm not sure if Linux supports it.
>>>>> Linux probably doesn't, save in the most recent kernels, and possibly
>>>>> only with appropriate patches.
I mostly concurr on this. Kernel support for yank-the-drive hot-plug for
libata is very new and has just been included in 2.6.18 (with the
appropriate updates to the Promise TX/TX2 SATA driver), but that's it so
far. Now that this is done though, most of the sata controllers that do
support this (ie, those not based on the ICH series) will probably follow
Warm-plug support, however, has been around for quite some time, and any 2.6
kernel that supports your sata controller should allow you to do this
(afaik)..more on this later.
>>>> In my case I'm wondering what could possibly go wrong? If the drive
>>>> is completely unmounted before it is powered down and removed, it
>>>> seems as though the OS has no reason to be concerned with how/when I
>>>> plug it in. Any ideas?
[ Snip doomsday scenarios and such ]
While experimenting with my system (two drive RAID1 w/trays, Promise SATAII
controller, kernel 2.6.15), yanking one of the drives while mounted and
active in the raid had no ill effects wrt. to the hardware or linux
subsystem. However, linux did not detect the removal of the drive and
bogged right down with IO errors as it constantly tried to continue updating
the missing hdd.
It did this forever (i would have assumed that it would have stopped trying
after a while and thrown the device out of the raid, but it did not).
Since there was nothing in the driver or libata to catch the drive-removed
interrupt (which did fire i noticed, as there was some sort of error about
an interrupt not being caught), this is not surprising and hopefully 2.6.18
will fix that. So this would handle having the device removed from the
kernel, but I'm still not sure what will happen to the raid subsystem if a
device suddenly disappears though.
>>> Also, if you don't stop the drive spinning before you pull it then you
>>> have cut power to a disk in rotation.
>> Huh. I guess unmounting the drive isn't enough to stop the spindle.
> No, though you can send the appropriate command sequences with sdparm or
> hdparm to put the drive to sleep. That should stop it well enough to
> help here.
You can also do this with manipulating /sys/.. and probably the 'scsiadd'
tool mentioned above.
I'm not sure what the issue is with stopping a drive in rotation, however??
If the drive isn't physically being moved while its spinning down, is that
really a problem?
>>>>>> Lastly, if there's any standard way to automate backup jobs (mounting
>>>>>> disks, rsync or whatever, unmounting, etc.) I'd appreciate a
>>>>>> reference. I can always use cron scripts but I imagine someone has
>>>>>> probably come up with something better.
>>>>> udev can fire off arbitrary code on insertion of a device. You can use
>>>>> that to trigger a script that will, basically, do all the work for you.
>> Are there "standard" scripts for this purpose, or will I be whipping
>> one up at home?
I haven't looked into this fully yet as to its applicability to a udev
system, but there is the 'scsirastools' package and its scripts
(http://scsirastools.sourceforge.net/), so that might give you something to
With my raid-1 system, for general ease (so my users do not need to interact
with any software), i do an automated cold-swap: I use ACPI w/the power
button to cleanly power down the server, swap the drive, then on bootup i
use the following script to add the degraded hdd into the array (md0 is made
up of sda1 and sdb1):
if ! (cat /proc/mdstat |grep md0 |grep -c sda) >/dev/null; then
mdadm /dev/md0 -a /dev/sda1;
if ! (cat /proc/mdstat |grep md0 |grep -c sdb) >/dev/null; then
mdadm /dev/md0 -a /dev/sdb1;
Once I upgrade to 2.6.18 I'm going to look into the scsirastools scripts in
depth, as hopefully they will at least provide a basis for automating the
entire process. Unfortunately, I think multiple layers of custom scripts
may still be required though.
[ Snip ]
> Yeah. I think, basically, SATA hot swap is still too new for me to want
> to use it in production. The USB case, for which hot swap is years old
> and well tested, is much more likely to be a success.
>>> Sorry if that wasn't clear to you -- the driver, OS and controller
>>> hardware need to be hot-swap capable for this to have a chance of
>>> working even remotely reliably.
>> Sounds like I've one out of three at the moment. Probably falling
>> back to USB is my best bet in the near term. I need to get the system
>> going and a backup system in place -- that's far more important than
>> having the backups be super fast.
I'm trying to get true yank-the-drive hotplug working with my system, but
i'm using this on a RAID1. David, since your system seems to be a simple
plug-in/bring-drive-up/rsync/bring-drive-down/un-plug, you might want to
look into using a warm-plug method:
1. plug drive in
2. manipulate /sys/ or use 'scsiadd' to rescan the scsi bus (udev should
pick up the new drive)
4. do your backing up
6. manipulate /sys/ or use 'scsiadd' to power down the drive and rescan the
scsi bus (udev should remove the drive's devnode)
7. unplug the drive
You would have to perform the commands interactively after plugging in the
drive and before unplugging it, but you certainly wouldn't need to bring
down the system to do it. And with 2.6.18 or higher (whenever your
controller's driver gets support) this process could be automated with
More information about the ubuntu-server