How to create a terabyte storage array?

[Anders Karlsson]

On 12/1/05, Ed Fletcher <ed at fletcher.ca> wrote:

[snip]

> I think that you are better off having the boot drive separate from
> the data drive/array.  Otherwise, you might not be able to boot if a
> drive on the array fails.  If your boot drive is separate, then you
> just replace the failed array drive and wait while it automatically
> rebuilds.

In an ideal situation, the OS should be completely separated from the
data. Different storage arrays, storage controller(s) etc. Ideal is a
HW RAID solution.

However, you can do this on just the five discs. In the installer,
five discs, all 200GB. Partition them all in a 2G/198G partition1/2
pattern. Mirror across the first partition on each disc (five way
mirror, md0) and make that your root filesystem (if you install a
server, 2GB is enough for all the tools, apps etc). RAID-5 across the
198GB partitions on each drive and you have a 996GB md1 device to
create you LVM PV/VG/LVs on. Any one disc fails, you still have four
copies of your OS, and a degraded but working storage array.

If you used 400GB discs, you would only need four of them. Same
principle applies. :)

> The boot drive doesn't have to be a large drive.  Especially if you
> don't load X.  I find that using a command line is good enough on a
> server and I save a lot of space by not using windows.  And almost
> everyone has a small, old drive sitting around that would be great in
> this environment.
>
> And while I'm on the subject of servers, I think that it's worth
> noting that you don't need much of a machine.  I have a raid 5 array
> at home running on a P-Pro 180.  It served up samba, ftp, apache (very
> light load) and I can burn cd's on it.  Never got the load up very
> high.  (Hmm . . . well it's still at home but not running.  The four
> 40 GB disks gave me 120 GB of raid 5.  I now have one 250 GB in an
> external USB case.  Uses a lot less electricity.)  Hmm . . . Another
> good point, get an extra power supply just for the array.  I found
> that I needed a second one in my server.

Power is always going to be the issue when involving lots of discs. It
would be preferable to use SCSI or SATA-2 (can tell the discs to spin
up one by one, easing the load on the PSU). It is worth mentioning
that dual redundant PSU's is a good idea and so is a UPS (make sure it
can run your system under full load for about ten minutes to give time
to shut down properly).

If I had the funds to do things the way I wanted, I'd have a SAN at
home with a FAStT700 storage controller and a blade center for
servers. One can always dream tho. :)

--
Anders Karlsson <trudheim at gmail.com>