[Bug 391975] Re: XFS incompatible with RAID0: Information get corrupted when Reading and Writting on Raid0 XFS partition over (N-1)*StripeSize [Only writes to N-1 stripes ovelaping info when writting to block over (N-1)*Size/N, also one stripe is

[Luis Lumbreras Picazo]

Sorry i can not because i use no more XFS, now i use EXT4 over Linux
Software Raid0 over three HDDs... no more XFS, also i have updated to last
Ubuntu

2010/8/13 Fabio Marconi <marconifabio at hotmail.it>

> Hello Luis Lumbreras Picazo
>
> Can you please attach here these files:
>
> /var/log/syslog
> /var/log/dmesg
> /var/log/kern.log
> /var/log/boot.log
>
> Thanks
>
> :)Fabio
>
> ** Package changed: ubuntu => linux (Ubuntu)
>
> --
> XFS incompatible with RAID0: Information get corrupted when Reading and
> Writting on Raid0 XFS partition over (N-1)*StripeSize [Only writes to N-1
> stripes ovelaping info when writting to block over (N-1)*Size/N, also one
> stripe is not used at read time]
> https://bugs.launchpad.net/bugs/391975
> You received this bug notification because you are a direct subscriber
> of the bug.
>
> Status in “linux” package in Ubuntu: Incomplete
>
> Bug description:
> UPDATED: Fail localized more specifically -> when writing / reading from
> Raid0 when XFS partition format.
> At read time one stripe is not used (maybe last?), so when you write to a
> block over (N-1)*Size/N the information on partition get corrupted, what i
> have not checked is where the info writted really goes, i think it also does
> not write on last stripe and info is overlaped, because with some little
> writes a los of directories, structure is corrupted.
>
> Since what i could see when only write to a block number lower than
> (N-1)*Size/N the information on partition do not get corrupted and can be
> read after correctly, but when write to a block number higher than
> (N-1)*Size/N the information on partition get corrupted, directory structure
> also get corrupted.
>
> Also reads seems to read only from N-1 stripes, because speed is always
> (N-1)/N when XFS over Raid0, but with EXT2/3/4 over same exact Raid0 the
> speed is very near to N/N. I mean if used N stripes, speed is
> N*SpeedOfStripe when EXT2/3/4, but with XFS it is (N-1)*SpeedOfStripe and
> since zcav reads at block level the partition format must not affect read
> speed because read is not at partition format level, it is at block level.
>
> Hope now is all clear. Have in mind whart i have done to ensure this:
> Install one and another ... times Ubuntu ... One per each
> test*configuration, too many hours, this night i have not dream just to make
> this checks... But if there is needed anything else... just ask me for them,
> ok?
>
> Now i also have been able to reproduce such on VirtualBox it would take
> less time to make more tests, since i can install Ubuntu, while checking on
> another Virtual...
>
> I have checked this with the next configurations (first three on real
> hardware, next on VirtualBOX):
>
> Two stripes:
> /dev/sda1 -> For Raid0
> /dev/sdb1 -> For Raid0
> /dev/md0 -> Composed of /dev/sda1 and /dev/sdb1, formatted as XFS
> RESULT of zcav -> Read speed is 1/2 of what must be, and 1/2 of what is if
> formatted as EXT4
>
> Three stripes:
> /dev/sda1 -> For Raid0
> /dev/sdb1 -> For Raid0
> /dev/sdc1 -> For Raid0
> /dev/md0 -> Composed of /dev/sda1, /dev/sdb1 and /dev/sdc1, formatted as
> XFS
> RESULT of zcav -> Read speed is 2/3 of what must be, and 2/3 of what is if
> formatted as EXT4
>
> Four stripes:
> /dev/sda1 -> For Raid0
> /dev/sdb1 -> For Raid0
> /dev/sdc1 -> For Raid0
> /dev/sdd1 -> For Raid0
> /dev/md0 -> Composed of /dev/sda1, /dev/sdb1, /dev/sdc1 and /dev/sdd1,
> formatted as XFS
> RESULT of zcav -> Read speed is 3/4 of what must be, and 3/4 of what is if
> formatted as EXT4
>
> This next ones only checked on VirtulBox:
> Five Stripes: Same idea, add /dev/sde1 to /dev/md0 with RESULT of zcav ->
> Read speed is 4/5 ...
> Six Stripes: Same idea, add /dev/sdf1 to /dev/md0 with RESULT of zcav ->
> Read speed is 5/6 ...
> Seven Stripes: Same idea, add /dev/sdg1 to /dev/md0 with RESULT of zcav ->
> Read speed is 6/7 ...
> Eight Stripes: Same idea, add /dev/sdh1 to /dev/md0 with RESULT of zcav ->
> Read speed is 7/8 ...
> Nine Stripes: Same idea, add /dev/sdi1 to /dev/md0 with RESULT of zcav ->
> Read speed is 8/9 ...
> Ten Stripes: Same idea, add /dev/sdj1 to /dev/md0 with RESULT of zcav ->
> Read speed is 9/10 ...
> ..etc..
> Up to 33 stripes (maximun i can with VirtualBox).
>
> In all cases it worked perfectly until a write on blocks with number higher
> to (N-1)*StripesSize, all writes to blocks with number higher to
> (N-1)*StripesSize makes the partition get corrupted.
>
> ------------------------------------------------------
> ----- I let below, the old text (with some little corrections just to
> correct missmatchs), so can be seen a little history how i got to this
> problem and so what have i check. ---
> ------------------------------------------------------
>
> I will try to explain as much detail as i can the BUG, because it is very
> critical, makes you to loose all data while working and do not let to
> recover anything, neither to reinstall without deleting all partitions on
> all HDDs.
>
> So, sorry me if this is a long text and if i repeat somethings, my
> intention is let as most clear as possible why it is Very Critical and how
> easy and severe is and how to be able to reproduce it
>
> All times i tried with XFS i got a non working OS when updating pakages.
>
> I mean: When updating packeges an error occurs and icons on desktop
> disappear, also bars icons changes, wallpaper disapear ... not able to open
> a new console, ... data on hdd and memory get corrupted and after reboot,
> obviously the system can not boot, system partition get totaly corrupted and
> unrecoverable.
>
> This error only happens (only tested with XFS and EXT4 by now) if the
> partition is XFS over Raid0, not if it is EXT4; rest types (EXT3, EXT2,
> ReiserFX, JFS, ... not tested, since with EXT4 worked fine and i did not
> have time to test).
>
> Let me try to explain how i have partitioned three HDDs (do not ask me why,
> i have my reasons and security for data is nor importat, i make external
> BackUPs of what i consider important, internal HDDs are only for OS and
> temporal data, so speed is top most important, more because i use huge file
> proccesing, fils bigger than 25GB each, and a lot of them):
>
> So this is how i partition my HDDs just to get the top most speed on boot
> and work, while having System separated from my PersonalDATA (all temporal,
> permanent data i store it on at least three different external usb hdds,
> "BackUPs"):
> /dev/sda:
> /dev/sda1 -> Radi0 stripe 0 for /
> /dev/sda2 -> Raid0 stripe 0 for /media/PersonalDATA
>
> /dev/sdb:
> /dev/sdb1 -> SWAP for SWAP
> /dev/sdb2 -> Raid0 stripe 1 for /
> /dev/sdb3 -> Raid0 stripe 1 for /media/PersonalDATA
>
> /dev/sdc:
> /dev/sdc1 -> ext3 for /boot
> /dev/sdc2 -> SWAP for SWAP
> /dev/sdc3 -> Raid0 stripe 2 for /
> /dev/sdc4 -> Raid0 stripe 2 for /media/PersonalDATA
>
> /dev/md0: Composed of /dev/sda1, /dev/sdb2 and /dev/sdc3 and formated as
> XFS, mount point /
> /dev/md1: Composed of /dev/sda2, /dev/sdb3 and /dev/sdc4 and formated as
> EXT3, mount point /media/PersonalDATA
>
> Sizes of partitions are as follows: /boot of 512MB, / of 24GB and SWAPs as
> big as needed just to equal size for /media/PersonalDATA on all three HDDs,
> the rest.
>
> Let me put an example how i calculate sizes on for example PATA HDD near
> 80GB, SATA HDDs 81GB so:
> /boot -> 512MB on SATA
> / -> 24GB striped along all three HDDs (Raid0)
> two SWAP partitions, one on each SATA
> PATA one 80GB - 24/3 -> 72GB, so size for /media/PersonalDATA is 72GB * 3 =
> 216GB
> One SATA 81GB - 512MB -> 80.5GB; 80.5GB - 24/3 -> 72.5GB; 72.5GB - 72GB ->
> 0.5 for SWAP
> Other SATA 81GB - 0MB -> 81GB; 81GB - 24/3 -> 73GB; 73GB - 72GB -> 1GB for
> SWAP
> Result:
> /boot -> 512MB
> SWAP1 -> 512MB
> SWAP2 -> 1024MB
> / -> 8GB*3 Raid0 -> 24GB
> /media/PersonalDATA -> 72GB*3 Raid0 -> 216GB
>
> Each HDDs reports on zcav quite similar speeds, about 57MB/s on fast part
> and 30MB/s on slower part
> Note: zcav report near 57MB/s*3->171MB/s (Really about 155MB/S) when not
> XFS is selected for partition
> Note: zcav report much less than 57MB/s*3->171MB/s (Really about 102MB/S)
> when XFS is selected for partition
> I think this indicates a big an serious problem.
>
> With this configuration system install from CD ends correctly, reboot and
> system works perfect.
>
> Since the error makes system unuseable any more and needed to reinstall
> from scracth i think it is a serious problem with XFS.
>
> WRONG -> Extra: On another machine with only two HDDs it works perfect
> Raid0 with XFS for system gives no problem (till now, hopes it will not give
> problems on future, but who knows?).
> RIGHT -> Extra: On another machine with only two HDDs it works perfect
> Raid0 with XFS for system gives no problem (till now, hopes it will not give
> problems on future, but who knows?), also get corrupted in same manner, it
> seemed to work fine because stripe size was bigger so problem occurs when
> more files where added, so a write to a block higher to (N-1)*StripeSize,
> since N=2, when write to the upper half of partition size.
>
> WRONG -> If i use Three HDDs it is unuseable, all times error occur when
> updating packages, why? i don't know.
> It is reproductable 100% on all PCs i have tested.
>
> RIGHT -> If i use Three HDDs it is unuseable, all times error occur when
> updating packages, why? because stripe size is so small that updating
> packages makes writes to last N stripe, so partition get corrupted.
> It is reproductable 100% on all PCs i have tested, also on Virtual ones
> under VirtualBOX.
>
> Let me explain how to reproduce this (allmost on three PCs i have tested it
> for four times on each and also on virtualBOX with a result of 100% of times
> the error becomes and system stop working ata same point, installing
> packages updates):
>
> Steps fllowed to reproduce it (quite WRONG, the RIGHT will be, make stripes
> sizes small enought, so last must be used):
> -Have a PC with on PATA HDD and three SATA HDDs of quite similar size
> -Delete all partitions on all (Three) HDDs [One PATA, Two SATA]
> -Create on PATA (/dev/sda) the next partitions: One for Raid0 for /
> (/dev/sda1) and another for Raid0 for /media/PersonalDATA (/dev/sda2)
> -Create on SATA (/dev/sdb) the next partitions: One for SWAP (/dev/sdb1),
> another for Raid0 for / (/dev/sdb2) and another for Raid0 for
> /media/PersonalDATA (/dev/sdb3)
> -Create on SATA (/dev/sdc) the next partitions: One for /boot (/dev/sdc1),
> another for SWAP (/dev/sdc2), another for Raid0 for / (/dev/sdc3) and
> another for Raid0 for /media/PersonalDATA (/dev/sdc4)
> -Create a Raid0 (/dev/md0) with /dev/sda1, /dev/sdb2 and /dev/sdc3 [for /]
> -Create a Raid0 (/dev/md1) with /dev/sda2, /dev/sdb3 and /dev/sdc4 [for
> /media/PersonalDATA]
> -Format /dev/md0 as XFS (that will cause problems after end installation,
> but if selected EXT4 no problem will be caused), mount point as /
> -Format /dev/md1 as EXT3, mount point as /media/PersonalDATA
> -Format /dev/sdc1 as EXT3, mount point as /boot
> -Format /dev/sdb1 as SWAP
> -Format /dev/sdc2 as SWAP
> -Install Ubuntu normally
>
> WRONG -> Installation ends without any problem, but now Update packages
> window appear.
> RIGHT -> Installation may end without any problem, but now Update packages
> window appear if Stripe size is small enought; if stripe is also smaller,
> also on installation corrupt occurs.
>
> WRONG -> Well if for / was selected XFS then Update packages will not end.
> RIGHT -> It is not Update packages what was causing the problem, is XFS, it
> get partition information corrupted when writing to blocks numbers higher
> than (N-1)*StripeSize/BlockSize
>
> What happened to me, since i let three stripes of 8GB each was just after
> downloading all packages, just at the begining of installing an error occurs
> while writing to XFS and the system crashes:
> -Icons on screen disapear, icons on bars changes, desktop wallpaper
> disapear, etc...
> -Trying to launch a console window fails, ...
> -If rebooted, the system does not reboot any more, so only solution is
> reinstalling from scratch again and again, until XFS for / is not selected
> (till now i only tested with XFS and EXT4).
>
> And it is also worst, after the error:
> If try to reinstall without previously reformating (deleting all
> partitions), at partitioning point on install it does not let you to
> continue because partitioning menus does not appear, you are forced to use
> Ctrl+Alt+F2 (alternative console) and use fdisk to delete partitions, then
> go back to install menus and tell it to redetect hard disk, only then it let
> you continue specifing partitions, installing, etc ...
>
> I consider this BUG very Critital, because it gets you with a non
> recoverable system, non bootable, non reinstallable and only solution after
> this fail is deleting all partitions on all HDDs with fdisk on console and
> not with install menus on screen.
>
> It seems to have a serious problem when writing to XFS over Raid0, while if
> using EXT4 no problem seems to appear.
> ADDED -> Problem occurs only when writing to blocks numbers higher than
> (N-1)*StripeSize/BlockSize
>
> Now it becomes to me one question:
> Would i need to re-install all other PCs that i have selected XFS for /? I
> will answer to this question by myself:
> Yes, of course, XFS is not reliable over Raid0, do not use XFS any more
> never again over Raid0 until solved, but what else can be used? EXT4? Yes,
> EXT4 is fast enought and works perfectly with Raid0, not only with two
> stripes, also with three stripes.
> Now, I do not need to test again speeds and select the top most fast except
> XFS, the selected option is EXT4, this would take me more than a week of
> intense test on some PCs, if anyone has done it...! Is there any reliable
> comparative? Of course i mean such test done over a Three HDDs on Radi0
> configuration.
>
> Something one time i have been able to see is that the fail seems to occur
> when writing to XFS over Raid0 when updating packages.
> More, zcav report makes me think something is very wrong with XFS, because
> read speed is 2/3 of what was supposed to be and if i use EXT4 instead of
> XFS i get with zcav a read speed of 3/3.
>
> VERIFICATED: Seems like if with XFS it would only be really using two HDDs,
> not all three, so any write corrupts data, but then why install works fine?
> and why rebooting works fine?, maybe corrupt only becomes whe crossing 2/3
> of total partition size?, so on install all is written correctly and on
> reboots most is read, writes are only for logs, etc.. so seems a fatal XFS
> incompatibility with Raid0.
>
> Note: I do not want to be asked why i use Three HDDs for Raid0, that is up
> to me and i have my reasons, let me expose just the top mot important for
> me:
> -I only have Two PATA channels and Two SATA channels.
> -One PATA channel has a DVD-Writer
> -The other PATA channel has a PATA HDD
> -I have two SATA ports, each with one HDD
> -So only Three HDDs can be at the same time working.  (Except i put another
> PCI controller that can not fit, no free PCI slots available)
> -I need top most speed on procesing huge files (bigger than 25GB each)
> So if i make a Raid0 only with two of them i get lower speeds, ...
> Since Raid0 works perfect with EXT3 and EXT4 partitions format and gets
> quite near full 3x speed, i configure Raid0 for using all Three HDDs.
>
> Why i do not use Raid5? Easy, i never let important files inside a PC, i
> only let inside PCs files for OS and only while using it the temporal work,
> so speed is much more important than data security.
> For data security i have another Three external USB HDDs, each with a full
> identical copy (BackUP), so if also two of them fails there is the third to
> be able to restore, and i never have powered on all at threee at the same
> time.
>
> Some important questions:
> Why Raid0 (/dev/md0) fails if XFS format is selected?, and why only when
> reinstalling packages?
> ANSWER -> Because XFS writes on Raid0 corrupt information when writes to
> blocks numbers higher than (N-1)*StripeSize/BlockSize
> Also, why zcav report 2/3 of the speed that must supposed to be?
> Also if instead of XFS i use EXT4, no problem and speed is 3/3.
>
> Tests done with zcav without changing partition scheme:
> -Data report of zcav for /dev/md0 over a Three HDDs Raid0 when formatted as
> XFS: 114MB/S
> (57MB/s * 3 = 171MB/s, what is wrong, why speed is 2/3? 57MB/s * 2 =
> 114MB/s ¡! Seems only to use two HDDs?)
> -Data report of zcav for /dev/md0 over a Three HDDs Raid0 when formatted as
> EXT4: 171MB/S
> (57MB/s * 3 = 171MB/S, it is ok)
>
> Hope i could let it enought clear.
>
> P.D.: Sorry not to attach zcav reports, but now i am not near the PCs (tee
> would let it, but i consider not much important)
>
> To unsubscribe from this bug, go to:
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/391975/+subscribe
>

-- 
XFS incompatible with RAID0: Information get corrupted when Reading and Writting on Raid0 XFS partition over (N-1)*StripeSize [Only writes to N-1 stripes ovelaping info when writting to block over (N-1)*Size/N, also one stripe is not used at read time]
https://bugs.launchpad.net/bugs/391975
You received this bug notification because you are a member of Kernel
Bugs, which is subscribed to linux in ubuntu.