Help, my disk array has one dead member

[Xen]

Karl Auer schreef op 25-03-2017 23:08:
> On Sat, 2017-03-25 at 13:59 +0100, Xen wrote:
>> Karl Auer schreef op 24-03-2017 4:02:
>> > If you go for RAID5, be aware that it is fairly slow to write. Very
>> > safe though, and good error recovery. RAID6 is even safer, and
>> > should be considered once you start to get much over 5 or 6
>> > terabytes, because the error rates even on modern drives start to
>> > make bit errors pretty much inevitable.
>> Normally sectors on drives (both 512 and 4k) are prefixed with error 
>> correction codes, aren't they?
> 
> Yes - and if the sector is gone, so is the error correction. That's why
> RAID stores error correction on different blocks, on different drives
> and (in the case of RAID6) multiple times. And the error correction in
> RAID covers multiple blocks, not just one, so you can recover a whole
> drive if you lose a RAID5 drive. RAID1 takes a different approach and
> just duplicates entire disks.
> 
>> That's the reason 4k is more efficient than 512.
> 
> I don't think so, but I'm not sure. I think the main reason larger
> blocks are more efficient is because larger single writes can be
> managed by the drive.
> 
>> How common is bit corruption during file transfer on say a regular
>> hard disk?
> 
> It's all about the statistical likelihood of an unrecoverable bit
> error, which drive manufacturers test and report as the bit error rate
> (BER). If you have a drive with a BER of 10^14 (the standard in
> consumer drives) it means that the manufacturer reports a likelihood of
> 1 in 10^14 that a bit will be read with the wrong value.
> 
> Read this for an influential article written by Rob Harris in 2007:
> 
> http://www.zdnet.com/article/why-raid-5-stops-working-in-2009/

What he says is that with high capacity drives there is a high 
likelihood of a unrecoverable error while reconstructing a failed drive. 
As the failed drive contained the redundancy for the other (2) drives, 
there is now no redundancy left and the error is now fatal, also because 
apparently reconstruction software will just fail on a single fault.

The point seems to be that you can't "disagree" with this guy if people 
are actually experiencing these rebuild failures. Everything else is 
conjecture, so it depends on what people actually report. Rational 
disagreement isn't going to help much.

> https://www.high-rely.com/blog/using-raid5-means-the-sky-is-falling/

"Such a failure could happen as a RAID-5 array is being rebuilt, 
striking a sector with a guaranteed URE on the parity disk happening at 
exactly 100,000,000,000,000 bits – unless it doesn’t."

This is nonsense. The original author never claimed that. Making such 
unfounded statements make me not feel very safe, I must say :p.

He has thus far not refuted the statistics, only saying that saying it 
is a guarantee is far off.

"or that all desktop hard disks will have read failure precisely at that 
10^14 bit"

This is nonsense, and he should know it. If there are these reported 
error rates for drives as manufacturors state, obviously it will be 
random, and moreoever it depends on the auto-correction not working. But 
that aside, if you actually did have 10^14 = ~12 TB then you could 
calculate the real probability that it would hit you while rebuilding 
say a 2TB disk.

This guy already done that, but the expected value of read errors (E) on 
a TB disk would simply be 2TB / 12.5 TB in that sense = 4/25 = 0.16 read 
errors per 2 TB.

Conversely the chance for at least 1 error is the converse of the chance 
of zero errors, which is (1 - 1 / 12.5) * 2 = (I mean power) = ... 1 - ( 
1 - 1 / 12500000000000 ) ^ 2000000000000 = some 14.8% chance of having a 
single read error which implies 85% chance of success. For one drive, 
and you need to read two.

Of course I have fixed my write errors in the above calculations but for 
one disk 14.8% failure comes out.

"All that is necessary to disprove this is the successful rebuilding of 
a RAID-5 set with 12TB or better capacity, as many primary and backup 
storage administrators have done countless times."

This is nonsense, but repeating it a hundred times would certainly give 
a good picture.

The only problems is that this author does not come up with any data 
himself, only the above statement.

So in an article meant to give "real world data" he gives barely any.

Anyway it is easy enough to test read rates on a 2TB disk, isn't it? The 
above author's primary concern is that the 10^14 number is incorrect, 
and more of a marketing figure than anything else.

Unrecoverable read errors should cause a DD to fail, for instance.

Currently reading a 10GB partition 4000 times lol. That isn't exactly 
the same, but that should be 40 runs of 2 TB. All md5sums should be 
identical.

That's on a NAS drive. Perhaps that is not so good for the drive in 
terms of wear :p.


> Basically the discussion starts with manufacturers reporting
> unrecoverable bit error rates of 1x10^14, or roughly 1 in 12.5
> terabytes read.

Yes. The author claims this is marketing gimmickry.

> The main thing to take away from the discussion IMHO is that RAID is no
> substitute for backup.


But if rebuilding RAIDS is really this fishy that defeats the whole 
point of it.

Moreoever a rebuild error may happen in unused space.

Rebuilds should not fail because of a single read error without more 
data given, but if they do, that is a really bad thing.