RAID configuration for large NAS

Question

I'm thinking of building a 24 1TB disk NAS box, but I'm not sure what the best drive configuration is. I'm looking at using the areca ARC-1280ML-2G controller, and hanging all 24 drives off of it.

I'd like it all to be mounted as one volume, due to the type of data we're storing on it. One crazy idea we had was to configure 6 4-disk RAID 5 volumes, then do software RAID 5 over those 6 volumes. That would mean any one volume could die on us and we'd still not lose data.

I should note that this is an R&D project, we've got an upcoming application where we'll be needing tens of terabytes of storage to be fast and highly available. But for the initial R&D phase we can accept some risk.

What is the best solution to this type of configuration? With 24 1 TB disks, it's likely that more than one will fail at the same time (or within the time it takes to rebuild the volume after a first failure), so I'm having trouble finding a good solution.

Answer 1

There is already a RAID level for what you want; it's called RAID 10.

The MTBF for professional and consumer level drives has increased by an order of magnitude in recent years, the uncorrectable error rate has stayed relatively constant. This rate is estimated at 10^14 bits, so one bit per 12 terabytes read, for consumer SATA drives, source.

So, for every scan of your passes of your 24Tb drive, statistically you will encounter at least 2 single bit errors. Each of those errors will trigger a RAID5 rebuild, and worse, during rebuild a second error will cause a double fault.

Answer 2

This is precisely my everyday work... building Linux storage servers.

• The Areca card is OK. You can use it in RAID-6, it will provide reasonable security. Buy the optional Battery backup unit, too.
• Use enterprise grade disks, not desktop drives. You'll spend 400 more bucks on your server, but it's well worth it. Buy two spare drives. Don't mess with it, use disks of the same model.
• For the filesystem, use XFS. Not kidding, ext3 and friends simply won't be up to the job for 16TB+ filesystems. Even in case of a serious crash, xfs_repair will be quite fast on a 20TB volume (15 minutes, no more).
• Preferably, use LVM2, it will ease the storage management, even if you don't plan to modify it much.
• install the areca management tool and write a cron job to send you a daily email with an health check.
• Don't forget backup. RAID is not a backup; if someone simply delete an important file, you won't be able to recover without a proper backup. I personnally use rdiff-backup to save all important data on a dedicated server with a one month history; you can also create two RAID volumes on your file server and backup one on the other.

Answer 3

wow, RAID5 over RAID5? Want to discuss performance problems? You'll have tons. The host you hang those off of will have kittens computing parity, writing that parity to 3 drives and then computing the parity of THAT parity and writing it to the 4th drive of that set. WOW!

Lets talk about RAID10. It's essentially RAID 1, but you divide up your drives in half and mirror that. It's fault tolerant in that you can lose 2 drives and still be fine, plus the performance is outstanding.

If you don't need an insane amount of space, but you've got a 24TB array sitting around with nothing better to do, but it absolutely positively has to be up, then you might consider RAID60. It's essentially RAID6 using mirrored sets of drives. You'll lose around half your drives, and performance will be bad, but you will be nearly guaranteed that the data will be there.

Really, I'd go with RAID10. It performs well and works fine. I second Evan's opinion that you probably shouldn't make giant RAID sets out of that many disks, because as he says, things like fsck and chkdsk will take forever, and but more importantly in my mind, because the statistical likelihood of a read error goes up as individual disk size does. I'd recommend 7-10 disks per set. You could create 3 very decently sized RAID volumes with that number of spindles.

Whichever you pick, remember to leave a couple of disks at hot spares, so that you can start rebuilding immediately, rather than having the array wait for you to replace them. As soon as a disk dies, the clock starts ticking for another one to go.

Answer 4

Why not RAID 1+0? It's all handled at the controller level...

Answer 5

I know that you said "R&D", but you also said "highly available". I would question the "savings" of a DIY solution versus purchasing off-the-shelf SAN gear to do this. When things go wrong with your DIY solution you're going to be in the unenviable position of having no one to contact for help. What does downtime cost you per hour? You can eat up the cost of some medium-tier SAN gear pretty quickly in downtime expense, ignoring the expense associated with out-right loss of data.

Regardless of what you do about the underlying disk, I wouldn't create a single filesystem that large.

Filesystem corruption is a real possibility (RAID controller issue, OS bugs, etc). In a volume that large, a filesystem check is going to take forever. I'd highly recommend using multiple volumes that can be logically combined to appear as a single filesytem (through various means-- you didn't mention the OS, so I can't give you specific ideas). If you have some filesystem corruption you will lose part of the logical volume, but you'll still be "up".

As an example: In a Windows world, running CHKDSK on a 20TB NTFS volume filled with files is going to be SLOW. In that kind of environment, I'd create multiple smaller NTFS volumes and logically combine them into a single namespace with DFS.

Answer 6

wazoox, answers are good I don't have the rep to give him more plus points, but I would add the following.

RAID 6 or at least 2 live parity discs per 10 discs, 16 at most that is if you can take around a day when performance will be impacted by your raid rebuild. If you can't live with the degradation then its going to have to be mirrored stripes.

If you are going the linux route, I would either use a hardware raid card ( with battery backup ) or have a raid controller in the disc enclosure. I agree that xfs is the filesystem of choice on Linux, however be aware that filesystems of around 50TB on xfs take more than 16GB of RAM if you need to run xfs_check.

I would seriously consider a good NAS box such as a NetApp as they are a lot less work long term, it depends how much your/the storage admins time is worth to the company.

Getting nfs/samba to perform well is a bit of a dark art, Are you going to be using 10GB ether or just aggregations of 1GB/sec ? ( Do not get Broadcomm cards especially 10GB ones ).

LVM2 is a no brainer but don't use the snap shotting as it is not fast.

Remember backups of this will take some time.

Test the way the system can fail before it goes into production and have it written up where you and your colleagues can find the docs when all goes wrong.

Answer 7

It depends on your read/write ratio. We use a lot of HP MSA70 external 25-disk SAS drive enclosures and always create them as a single RAID6 array as our read to write ratio is 99%:1% so we don't care that R6 is about the slowest at writing (still pretty quick, just not that good compared to others). This way we have 23 disks worth of data available to us, have very good, as in VERY good, random read and overall read bandwidth benefits and can survive two disk failures.

As a rough guide a RAID5 array shouldn't have more than about 14 disks in one array, whilst a RAID6 should be fine with up to 54 disks or so - obviously the bigger the array the bigger the gulf between read and write performance and the slower rebuilds will take but it CAN be a good trade-off.

Answer 8

I would add two standby disks for a start.

RAID 5 or 6 is OK for random reads or large sequential reads and writes. If you are going to get lots of small writes go with RAID 10 since RAID 5+ takes a 4 x hit on small writes.

If you're going to turn on write cache remember to back it with a battery.