Corrupted ext3 filesystem on top of Adaptec RAID5 array

[Michael R. Head]

Woke up this morning to find one of my dapper servers in the midst of a
crisis. The filesystem had been remounted readonly and large parts are
corrupted. The machine is fairly new, and a backup regime has not been
put in place yet.

I'm curious about a few things:

1) How did this happen? The logs appear to have been destroyed along
with most any diagnostic information. (syslog is zeroed out)
2) What's the best way to recover?
3) Has anyone seen something similar?
4) How can I prevent this in the future?

Here are some symptoms:
1) The filesystem remounted read only due to some error I can't discern
2) several programs simply fail to run:
burner at rhea:~$ mount
-bash: /bin/mount: Input/output error
burner at rhea:~$ lspci
-bash: /usr/bin/lspci: Input/output error
burner at rhea:~$ dmesg
-bash: /bin/dmesg: Input/output error
3) /var/log/syslog was corrupted and its contents zeroed out:
burner at rhea:~$ ls -l /var/log/syslog
-rw-r----- 1 root adm 3699 2006-07-10 14:10 /var/log/syslog
burner at rhea:~$ less /var/log/syslog
"/var/log/syslog" may be a binary file.  See it anyway?
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^
4) Many files are in a bizarre state:
burner at rhea:~$ ls -l /usr/bin |grep -v rw |head
total 162456
?---------  ? ?      ?             ?                ? /usr/bin/411toppm
?---------  ? ?      ?             ?                ? /usr/bin/a2p
?---------  ? ?      ?             ?                ? /usr/bin/a2ping
?---------  ? ?      ?             ?                ? /usr/bin/afm2tfm
?---------  ? ?      ?             ?                ? /usr/bin/afmdiff.awk
?---------  ? ?      ?             ?                ? /usr/bin/allcm
?---------  ? ?      ?             ?                ? /usr/bin/allneeded
?---------  ? ?      ?             ?                ? /usr/bin/anytopnm
?---------  ? ?      ?             ?                ? /usr/bin/asciitopgm

I'm baffled, but I do have a possible cause.
Recently I resized (shrunk) the root filesystem to make more room for
swap. For this, I used the desktop CD with gparted to shrink the
filesystem and partition for '/' and increase the swap partition.
Is it possible that this caused the failure?

mike
-- 
Michael R. Head <burner at suppressingfire.org>
suppressingfire.org
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 191 bytes
Desc: This is a digitally signed message part
URL: <https://lists.ubuntu.com/archives/ubuntu-users/attachments/20060711/944f2566/attachment.sig>