[Bug 391975] [NEW] XFS incompatible with RAID0: Information get corrupted when Reading and Writting on Raid0 XFS partition over (N-1)*StripeSize [Only writes to N-1 stripes ovelaping info when writting to block over (N-1)*Size/N, also one stripe is not used at read time]

[Launchpad Bug Tracker]

You have been subscribed to a public bug:

UPDATED: Fail localized more specifically -> when writing / reading from Raid0 when XFS partition format.
At read time one stripe is not used (maybe last?), so when you write to a block over (N-1)*Size/N the information on partition get corrupted, what i have not checked is where the info writted really goes, i think it also does not write on last stripe and info is overlaped, because with some little writes a los of directories, structure is corrupted.

Since what i could see when only write to a block number lower than
(N-1)*Size/N the information on partition do not get corrupted and can
be read after correctly, but when write to a block number higher than
(N-1)*Size/N the information on partition get corrupted, directory
structure also get corrupted.

Also reads seems to read only from N-1 stripes, because speed is always
(N-1)/N when XFS over Raid0, but with EXT2/3/4 over same exact Raid0 the
speed is very near to N/N. I mean if used N stripes, speed is
N*SpeedOfStripe when EXT2/3/4, but with XFS it is (N-1)*SpeedOfStripe
and since zcav reads at block level the partition format must not affect
read speed because read is not at partition format level, it is at block
level.

Hope now is all clear. Have in mind whart i have done to ensure this:
Install one and another ... times Ubuntu ... One per each
test*configuration, too many hours, this night i have not dream just to
make this checks... But if there is needed anything else... just ask me
for them, ok?

Now i also have been able to reproduce such on VirtualBox it would take
less time to make more tests, since i can install Ubuntu, while checking
on another Virtual...

I have checked this with the next configurations (first three on real
hardware, next on VirtualBOX):

Two stripes:
/dev/sda1 -> For Raid0
/dev/sdb1 -> For Raid0
/dev/md0 -> Composed of /dev/sda1 and /dev/sdb1, formatted as XFS
RESULT of zcav -> Read speed is 1/2 of what must be, and 1/2 of what is if formatted as EXT4

Three stripes:
/dev/sda1 -> For Raid0
/dev/sdb1 -> For Raid0
/dev/sdc1 -> For Raid0
/dev/md0 -> Composed of /dev/sda1, /dev/sdb1 and /dev/sdc1, formatted as XFS
RESULT of zcav -> Read speed is 2/3 of what must be, and 2/3 of what is if formatted as EXT4

Four stripes:
/dev/sda1 -> For Raid0
/dev/sdb1 -> For Raid0
/dev/sdc1 -> For Raid0
/dev/sdd1 -> For Raid0
/dev/md0 -> Composed of /dev/sda1, /dev/sdb1, /dev/sdc1 and /dev/sdd1, formatted as XFS
RESULT of zcav -> Read speed is 3/4 of what must be, and 3/4 of what is if formatted as EXT4

This next ones only checked on VirtulBox:
Five Stripes: Same idea, add /dev/sde1 to /dev/md0 with RESULT of zcav -> Read speed is 4/5 ...
Six Stripes: Same idea, add /dev/sdf1 to /dev/md0 with RESULT of zcav -> Read speed is 5/6 ...
Seven Stripes: Same idea, add /dev/sdg1 to /dev/md0 with RESULT of zcav -> Read speed is 6/7 ...
Eight Stripes: Same idea, add /dev/sdh1 to /dev/md0 with RESULT of zcav -> Read speed is 7/8 ...
Nine Stripes: Same idea, add /dev/sdi1 to /dev/md0 with RESULT of zcav -> Read speed is 8/9 ...
Ten Stripes: Same idea, add /dev/sdj1 to /dev/md0 with RESULT of zcav -> Read speed is 9/10 ...
..etc..
Up to 33 stripes (maximun i can with VirtualBox).

In all cases it worked perfectly until a write on blocks with number
higher to (N-1)*StripesSize, all writes to blocks with number higher to
(N-1)*StripesSize makes the partition get corrupted.

------------------------------------------------------
----- I let below, the old text (with some little corrections just to correct missmatchs), so can be seen a little history how i got to this problem and so what have i check. ---
------------------------------------------------------

I will try to explain as much detail as i can the BUG, because it is
very critical, makes you to loose all data while working and do not let
to recover anything, neither to reinstall without deleting all
partitions on all HDDs.

So, sorry me if this is a long text and if i repeat somethings, my
intention is let as most clear as possible why it is Very Critical and
how easy and severe is and how to be able to reproduce it

All times i tried with XFS i got a non working OS when updating pakages.

I mean: When updating packeges an error occurs and icons on desktop
disappear, also bars icons changes, wallpaper disapear ... not able to
open a new console, ... data on hdd and memory get corrupted and after
reboot, obviously the system can not boot, system partition get totaly
corrupted and unrecoverable.

This error only happens (only tested with XFS and EXT4 by now) if the
partition is XFS over Raid0, not if it is EXT4; rest types (EXT3, EXT2,
ReiserFX, JFS, ... not tested, since with EXT4 worked fine and i did not
have time to test).

Let me try to explain how i have partitioned three HDDs (do not ask me
why, i have my reasons and security for data is nor importat, i make
external BackUPs of what i consider important, internal HDDs are only
for OS and temporal data, so speed is top most important, more because i
use huge file proccesing, fils bigger than 25GB each, and a lot of
them):

So this is how i partition my HDDs just to get the top most speed on boot and work, while having System separated from my PersonalDATA (all temporal, permanent data i store it on at least three different external usb hdds, "BackUPs"):
/dev/sda:
/dev/sda1 -> Radi0 stripe 0 for /
/dev/sda2 -> Raid0 stripe 0 for /media/PersonalDATA

/dev/sdb:
/dev/sdb1 -> SWAP for SWAP
/dev/sdb2 -> Raid0 stripe 1 for /
/dev/sdb3 -> Raid0 stripe 1 for /media/PersonalDATA

/dev/sdc:
/dev/sdc1 -> ext3 for /boot
/dev/sdc2 -> SWAP for SWAP
/dev/sdc3 -> Raid0 stripe 2 for /
/dev/sdc4 -> Raid0 stripe 2 for /media/PersonalDATA

/dev/md0: Composed of /dev/sda1, /dev/sdb2 and /dev/sdc3 and formated as XFS, mount point /
/dev/md1: Composed of /dev/sda2, /dev/sdb3 and /dev/sdc4 and formated as EXT3, mount point /media/PersonalDATA

Sizes of partitions are as follows: /boot of 512MB, / of 24GB and SWAPs
as big as needed just to equal size for /media/PersonalDATA on all three
HDDs, the rest.

Let me put an example how i calculate sizes on for example PATA HDD near 80GB, SATA HDDs 81GB so:
/boot -> 512MB on SATA
/ -> 24GB striped along all three HDDs (Raid0)
two SWAP partitions, one on each SATA
PATA one 80GB - 24/3 -> 72GB, so size for /media/PersonalDATA is 72GB * 3 = 216GB
One SATA 81GB - 512MB -> 80.5GB; 80.5GB - 24/3 -> 72.5GB; 72.5GB - 72GB -> 0.5 for SWAP
Other SATA 81GB - 0MB -> 81GB; 81GB - 24/3 -> 73GB; 73GB - 72GB -> 1GB for SWAP
Result:
/boot -> 512MB
SWAP1 -> 512MB
SWAP2 -> 1024MB
/ -> 8GB*3 Raid0 -> 24GB
/media/PersonalDATA -> 72GB*3 Raid0 -> 216GB

Each HDDs reports on zcav quite similar speeds, about 57MB/s on fast part and 30MB/s on slower part
Note: zcav report near 57MB/s*3->171MB/s (Really about 155MB/S) when not XFS is selected for partition
Note: zcav report much less than 57MB/s*3->171MB/s (Really about 102MB/S) when XFS is selected for partition
I think this indicates a big an serious problem.

With this configuration system install from CD ends correctly, reboot
and system works perfect.

Since the error makes system unuseable any more and needed to reinstall
from scracth i think it is a serious problem with XFS.

WRONG -> Extra: On another machine with only two HDDs it works perfect Raid0 with XFS for system gives no problem (till now, hopes it will not give problems on future, but who knows?).
RIGHT -> Extra: On another machine with only two HDDs it works perfect Raid0 with XFS for system gives no problem (till now, hopes it will not give problems on future, but who knows?), also get corrupted in same manner, it seemed to work fine because stripe size was bigger so problem occurs when more files where added, so a write to a block higher to (N-1)*StripeSize, since N=2, when write to the upper half of partition size.

WRONG -> If i use Three HDDs it is unuseable, all times error occur when updating packages, why? i don't know. 
It is reproductable 100% on all PCs i have tested.

RIGHT -> If i use Three HDDs it is unuseable, all times error occur when updating packages, why? because stripe size is so small that updating packages makes writes to last N stripe, so partition get corrupted. 
It is reproductable 100% on all PCs i have tested, also on Virtual ones under VirtualBOX.

Let me explain how to reproduce this (allmost on three PCs i have tested
it for four times on each and also on virtualBOX with a result of 100%
of times the error becomes and system stop working ata same point,
installing packages updates):

Steps fllowed to reproduce it (quite WRONG, the RIGHT will be, make stripes sizes small enought, so last must be used):
-Have a PC with on PATA HDD and three SATA HDDs of quite similar size
-Delete all partitions on all (Three) HDDs [One PATA, Two SATA]
-Create on PATA (/dev/sda) the next partitions: One for Raid0 for / (/dev/sda1) and another for Raid0 for /media/PersonalDATA (/dev/sda2)
-Create on SATA (/dev/sdb) the next partitions: One for SWAP (/dev/sdb1), another for Raid0 for / (/dev/sdb2) and another for Raid0 for /media/PersonalDATA (/dev/sdb3)
-Create on SATA (/dev/sdc) the next partitions: One for /boot (/dev/sdc1), another for SWAP (/dev/sdc2), another for Raid0 for / (/dev/sdc3) and another for Raid0 for /media/PersonalDATA (/dev/sdc4)
-Create a Raid0 (/dev/md0) with /dev/sda1, /dev/sdb2 and /dev/sdc3 [for /]
-Create a Raid0 (/dev/md1) with /dev/sda2, /dev/sdb3 and /dev/sdc4 [for /media/PersonalDATA]
-Format /dev/md0 as XFS (that will cause problems after end installation, but if selected EXT4 no problem will be caused), mount point as /
-Format /dev/md1 as EXT3, mount point as /media/PersonalDATA
-Format /dev/sdc1 as EXT3, mount point as /boot
-Format /dev/sdb1 as SWAP
-Format /dev/sdc2 as SWAP
-Install Ubuntu normally

WRONG -> Installation ends without any problem, but now Update packages window appear.
RIGHT -> Installation may end without any problem, but now Update packages window appear if Stripe size is small enought; if stripe is also smaller, also on installation corrupt occurs.

WRONG -> Well if for / was selected XFS then Update packages will not end.
RIGHT -> It is not Update packages what was causing the problem, is XFS, it get partition information corrupted when writing to blocks numbers higher than (N-1)*StripeSize/BlockSize

What happened to me, since i let three stripes of 8GB each was just after downloading all packages, just at the begining of installing an error occurs while writing to XFS and the system crashes:
-Icons on screen disapear, icons on bars changes, desktop wallpaper disapear, etc... 
-Trying to launch a console window fails, ... 
-If rebooted, the system does not reboot any more, so only solution is reinstalling from scratch again and again, until XFS for / is not selected (till now i only tested with XFS and EXT4).

And it is also worst, after the error:
If try to reinstall without previously reformating (deleting all partitions), at partitioning point on install it does not let you to continue because partitioning menus does not appear, you are forced to use Ctrl+Alt+F2 (alternative console) and use fdisk to delete partitions, then go back to install menus and tell it to redetect hard disk, only then it let you continue specifing partitions, installing, etc ...

I consider this BUG very Critital, because it gets you with a non
recoverable system, non bootable, non reinstallable and only solution
after this fail is deleting all partitions on all HDDs with fdisk on
console and not with install menus on screen.

It seems to have a serious problem when writing to XFS over Raid0, while if using EXT4 no problem seems to appear.
ADDED -> Problem occurs only when writing to blocks numbers higher than (N-1)*StripeSize/BlockSize

Now it becomes to me one question: 
Would i need to re-install all other PCs that i have selected XFS for /? I will answer to this question by myself: 
Yes, of course, XFS is not reliable over Raid0, do not use XFS any more never again over Raid0 until solved, but what else can be used? EXT4? Yes, EXT4 is fast enought and works perfectly with Raid0, not only with two stripes, also with three stripes.
Now, I do not need to test again speeds and select the top most fast except XFS, the selected option is EXT4, this would take me more than a week of intense test on some PCs, if anyone has done it...! Is there any reliable comparative? Of course i mean such test done over a Three HDDs on Radi0 configuration.

Something one time i have been able to see is that the fail seems to occur when writing to XFS over Raid0 when updating packages.
More, zcav report makes me think something is very wrong with XFS, because read speed is 2/3 of what was supposed to be and if i use EXT4 instead of XFS i get with zcav a read speed of 3/3.

VERIFICATED: Seems like if with XFS it would only be really using two
HDDs, not all three, so any write corrupts data, but then why install
works fine? and why rebooting works fine?, maybe corrupt only becomes
whe crossing 2/3 of total partition size?, so on install all is written
correctly and on reboots most is read, writes are only for logs, etc..
so seems a fatal XFS incompatibility with Raid0.

Note: I do not want to be asked why i use Three HDDs for Raid0, that is up to me and i have my reasons, let me expose just the top mot important for me:
-I only have Two PATA channels and Two SATA channels.
-One PATA channel has a DVD-Writer
-The other PATA channel has a PATA HDD
-I have two SATA ports, each with one HDD
-So only Three HDDs can be at the same time working.  (Except i put another PCI controller that can not fit, no free PCI slots available)
-I need top most speed on procesing huge files (bigger than 25GB each)
So if i make a Raid0 only with two of them i get lower speeds, ... 
Since Raid0 works perfect with EXT3 and EXT4 partitions format and gets quite near full 3x speed, i configure Raid0 for using all Three HDDs.

Why i do not use Raid5? Easy, i never let important files inside a PC, i only let inside PCs files for OS and only while using it the temporal work, so speed is much more important than data security.
For data security i have another Three external USB HDDs, each with a full identical copy (BackUP), so if also two of them fails there is the third to be able to restore, and i never have powered on all at threee at the same time.

Some important questions:
Why Raid0 (/dev/md0) fails if XFS format is selected?, and why only when reinstalling packages? 
ANSWER -> Because XFS writes on Raid0 corrupt information when writes to blocks numbers higher than (N-1)*StripeSize/BlockSize
Also, why zcav report 2/3 of the speed that must supposed to be? 
Also if instead of XFS i use EXT4, no problem and speed is 3/3.

Tests done with zcav without changing partition scheme:
-Data report of zcav for /dev/md0 over a Three HDDs Raid0 when formatted as XFS: 114MB/S 
(57MB/s * 3 = 171MB/s, what is wrong, why speed is 2/3? 57MB/s * 2 = 114MB/s ¡! Seems only to use two HDDs?)
-Data report of zcav for /dev/md0 over a Three HDDs Raid0 when formatted as EXT4: 171MB/S 
(57MB/s * 3 = 171MB/S, it is ok)

Hope i could let it enought clear.

P.D.: Sorry not to attach zcav reports, but now i am not near the PCs
(tee would let it, but i consider not much important)

** Affects: linux (Ubuntu)
     Importance: Undecided
         Status: Incomplete


** Tags: critical
-- 
XFS incompatible with RAID0: Information get corrupted when Reading and Writting on Raid0 XFS partition over (N-1)*StripeSize [Only writes to N-1 stripes ovelaping info when writting to block over (N-1)*Size/N, also one stripe is not used at read time]
https://bugs.launchpad.net/bugs/391975
You received this bug notification because you are a member of Kernel Bugs, which is subscribed to linux in ubuntu.