8.04-1 won't boot from degraded raid

Tue Aug 26 21:56:12 UTC 2008

On Tue, Aug 26, 2008 at 01:56:24PM -0600, Sam Howard wrote:
> I really don't want to get into the middle of a flame war, 

Yes, sorry about that.

> but I don't understand something you wrote and would like
> clarification so that I am not assuming something incorrectly.

Certainly.

>> Imagine a scenario where the disk controller is flaky. Disk A goes
>> away while the system is running, and is then out of date. You reboot
>> the machine (or perhaps it rebooted itself because the flaky
>> controller short circuited or whatever), and for whatever reason
>> (flaky controller, remember?), the system boots from disk B instead.
>> The changes to your filesystem since disk A disappeared away are not
>> there, and new changes are being written to disk B, and there's no
>> chance of merging the two.  This is what I refer to as "having a very
>> bad day".
> I agree that data is is of the utmost importance, but in your
> scenerio, you loose disk A in a running system, but you imply that
> upon reboot on disk B, your data between A and B is not in sync.  It
> is no more out of sync than when the system was running with a broken
> A disk anyway.  

Correct.

> I am assuming you are talking about RAID1, which would keep the disks
> in sync until one of them goes away, at which point, B is your current
> disk anyway.

Right.

> "The changes to your filesystem since disk A disappeared away are not
> there, and new changes are being written to disk B, and there's no
> chance of merging the two."
> 
> Did you just mistype your example, or am I missing something really obvious
> here?

I certainly did. Thanks for the correction. What I meant was this:

Imagine a scenario where the disk controller is flaky. Disk B goes away
while the system is running, and is then out of date. You reboot the
machine (or perhaps it rebooted itself because the flaky controller
short circuited or whatever), and for whatever reason (flaky controller,
remember?), the system now boots from disk B instead. The changes to
your filesystem since disk B disappeared away are not there, and new
changes are being written to disk B, and there's no chance of merging
the two.

> Just to muddy the waters a bit more about which boot-on-broken-raid
> function is more useful, I have to vote on booting on 1 disk of a
> broken raid.  I say this for a few reasons:
> 
> 1 - since I run RAID1, my disks are always in sync (or the broken disk is
> broken and out of sync and needs to be replaced anyway)

You're making an assumption here: You're assuming that once a disk
fails, it vanishes completely and is never to be heard of again.  This
is, unfortunately, not the case. If a flaky controller makes disk A come
and go, the running system is aware of this and will have taken it out
of the RAID array. When the system boots, it doesn't know that disk A
was acting up and that disk B is the good one, so it'll happily boot
from either disk. If your faulty controller now only sees disk A
(hardware failures are not famous for their deterministic features),
it'll boot from that, and hell breaks loose.

> 2 - I expect to be alerted by the mdadm daemon when a disk goes broken, so I
> should know I have something to go fix (note: make sure the mdadm is
> configured to send e-mail to someone who will actually *see* it)

Good point. However, if you replace it before rebooting, you will not be
left with a degraded raid set the next time you boot, right?

> 3 - most of my servers are remote, so the ability to affect a repair and
> recovery w/o a booting system (albeit on the surviving disk) is between slim
> and none ... if you've ever tried to talk a non-technical user through
> booting on a live cd and then configure the networking, you know what I'm
> talking about!

I hear you. All my servers are, in fact, remote. I'm however in the
happy situation that if a machine fails to come online after a reboot, I
can boot up a RAM-based rescue system from whence I can diagnose the
system. I realise you might not be as fortunate.

Notwithstanding, I'd still prefer the system not just boot without any
sort of interaction with an admin of some sort. A simple dialog asking
if you understand the risks involved and still want to continue booting
would be perfectly acceptable. That would make the required guidance
much simpler.  

> Specifically, I am working on a system about 2,000 miles away from me,
> trying to recover to a new disk ... were I not able to boot off of the
> surviving disk, we would be talking about FedEx'ing a server to me to
> try to boot off of CD (after installing a CD drive, of course) or
> network, replace and repair, and the FedEx back.  Seems sort of silly,
> doesn't it?  It also opens the door for additional damage or data loss
> during shipping.

Even though Michael seems to enjoy making it appear otherwise, I'm
perfectly aware of your use case, and fully acknowledge its validity,
and I'm very happy that Dustin has put a lot of effort into making this
possible, so that you won't have to go through that hassle if your
priorities are different than mine.

> I support (professionally) servers literally around the world, of many
> *nix operating systems, and the ability to remotely recover a server
> is paramount.

I understand. The perfect solution for me would be an ssh server in the
initramfs so that I could ssh into the server and take a look around,
reassure myself that the faulty disk has been properly identified, etc.,
etc. and then take appropriate action.

> Ironically, the server I am recovering now is an old Debian server
> that I have build Hardy replacements for, but now I am a bit nervous
> about sending the replacement servers into the field.  I would very
> much like to have a workaround or fix that allows me to remotely
> repair a Hardy server ...  especially since most everything else seems
> to work so nicely in Hardy (Xen server up in <30 minutes and only a
> handful of apt-gets and such before hosting guests!).

This is all very valuable input. It's good to have some leverage when we
engage in the discussion about perhaps getting this functionality pushed
back into hardy.

-- 
Soren Hansen               | 
Virtualisation specialist  | Ubuntu Server Team
Canonical Ltd.             | http://www.ubuntu.com/