14

I wonder what is the fastest and safest RAID combination for SATA drives and general use (some write, mostly read)?

RAID 0 is fast but utterly unsafe, RAID 1 is safe but slow, RAID 5 is safe but not so fast, especially on the cheap controllers (XOR calculations).

It seem that RAID 1+0 or RAID 10 is the best combination. You get mirroring for safety and striping for speed. Are there any other best or more optimal combinations? The only drawback of the RAID 10 is inefficient storage utilization.

6
  • How many disks do you plan to have in your array? Will you do mostly reads or writes? What kinds of reads/writes are we talking about? (My point: No, there are no magical RAID NN choice perfect for any and all situations.)
    – andol
    Mar 10, 2010 at 6:15
  • Well I have update the question. I am interested in SATA drives. Use will be general some read some write, but mostly read. I agree with you that there are no magical RAID combination, but anyway there exist optimal solution in what I am interested in. Mar 10, 2010 at 6:49
  • It's not the calculation in RAID5 that is slow. It's the fact that is has to read and write for every write.
    – ptman
    Mar 10, 2010 at 7:19
  • ptman: this is also a factor, yes. Mar 10, 2010 at 8:03
  • How many disks will be a large factor in determining this as well... and RAID is for getting better uptime - it is NOT for protecting your data. That's what you must have backups for. Mar 10, 2010 at 9:13

3 Answers 3

12

It also depends upon the number of drives : with 4 drives, go for RAID-10. With more than 8 drives, RAID-6 will probably be fast enough with a good RAID controller (3Ware, Areca, Intel 52xxx series). Here are the numbers :

  • 4 x 1TB, RAID 10 : 2TB available space, 180 MB/s write, 190 MB/s read
  • 8 x 1TB, RAID 10 : 4TB available space, 360 MB/s write, 400 MBs read
  • 8 x 1TB, RAID-5 (dangerous): 7 TB available, 420 MB/s write, 440 MB/s read (3Ware)
  • 8 x 1TB, RAID-6 : 6 TB available, 240 MB/s write, 360 MB/s read (3Ware)
  • 16 x 1TB, RAID-6 : 14TB available, 280 MB/s write, 700 MB/s read (3Ware)

As you can see, with about 8 drives RAID 5 and RAID 6 are quite competitive in sequential performance with RAID-10 (not so with a shitty card such as Promise, etc). Write performance is quite limited in RAID-6, though tolerable given enough drives.

With big drives, RAID-5 is relatively unsafe because of the long time (3 to 4 hours, up to 7 to 8 hours) necessary for rebuilding. You may go to RAID-5 with 6 or 8 drives though, but you must stop all write operations in case of a drive failure until the array is rebuilt. This way it's "safe enough".

Also, don't use desktop drives in a RAID array with more than 4 drives. Vibrations and read errors will kill performance.

5
  • 2
    RAID 5 is also unsafe because of unrecoverable read errors... Mar 10, 2010 at 13:05
  • I've set up several hundred servers using RAID-5 and RAID-6, and unrecoverable RAID errors are extremely rare, sufficiently so not to be a serious trouble.
    – wazoox
    Mar 10, 2010 at 17:30
  • 2
    Sucky drives. 4x300gb, RAID 10, 600gb available - 500mb/s COPY (read+write same time, same drive set). Adaptec 5805 and... Velociratpros (enterprise edition, not the ones mounted in 3.5" coolers that dont do anything). Never looked back ;) I second the warning for desktop drives - the bearings are different on enterprise drives, made for more vibrations (which are common when you pack up 10 or 20 or more drives in a cage).
    – TomTom
    Mar 10, 2010 at 20:00
  • 8 x 1TB, RAID-6 : 6 TB available, 240 MB/s write Way late here, but this looks like large-block streaming sequential write numbers and not small-block random write operations like a real-world file system would do as a large part of its workload. Small-block random write operations to a RAID5/6 arrays can drive throughput down into kBs/sec range. Dec 5, 2022 at 13:31
  • @AndrewHenle it depends really. The unavoidable small block writes such as metadata activity are efficiently cached by modern RAID controllers or RAID software. Continuous 4k random writes of a RAID-6 array are more or less in the range of a single drive performance (because it's necessary to read the full stripe for every synced write to rewrite parities). But nobody's using RAID5/6 this way anymore (random access = SSD anyway).
    – wazoox
    Dec 6, 2022 at 14:26
9

There pretty much is nothing better than RAID 10 for speed. Point - because you get write decoupling. Any more efficient RAID (5, 6) has a bottleneck in writing that is higher than RAID 10.

That said, you MAY get away replacing a RAID 10 normal dsics with a RAID 5 or RAID 6 based on SSD's - which may not be that much more expensive thanks to the need to have less discs.

Raid 5 gets unsafe with too large / too many discs - in this case you need to go Raid 6. Problem is that if a disc fails in Raid 5.... at a certain point you are more or less likely to get a second disc failure DURING THE REBUILD, at which point the Raid fails. The limit is currently seen around 2gb discs, so more relevant for archive setups. Raid 6 solves that for now.

Personally I currently go Raid 5/6 for storage, file servers. Raid 10 for virthal server operating system discs (but then I ahve like 6-10 platters and run 40 or so servers off that - if they all boot, that is pretty much disc hell) an RAID 10 for some database data areas.

Another thing to look at is the discs you use. higher IOPS are better. Cheap would be normal SATA discs, high end are 15000 RPM SAS discs that cost a fortune. The Western Digical Velociraptor 2.5" enterprise version is a good medium ground - 300gb per disc, 10000RPM. About double the IO of a standard SATA disc, but a LOT cheaper than SAS high end discs. But then, a RAID 5 of SSD's soonish kills those in performance AND price... because you need less.

As andol said, it all depends on your needs. What you try to do?

And finally - this is not SATA depending at all. Actualy thanks to SAS interoperability with SATA you can plug any SATA drive into a SAS backbone (they are compatible - even physically) and use the SAS infrastructure.

4
  • "The limit is currently seen around 2gb discs" - do you mean 2TB?
    – ptman
    Mar 10, 2010 at 7:16
  • sorry, yes- 2tb.
    – TomTom
    Mar 10, 2010 at 7:58
  • I second the RAID 5 issue with unrecoverable read errors. It SUCKS to discover that two hours into a rebuild and having to start with a fresh recovery from backup! ARGH! And while RAID 6/10 solves that, I've read some grumblings that as data needs continue to increase, those will soon have issues too. Data density just keep jumping higher and with it comes more issues with data integrity and reliability. Mar 10, 2010 at 13:10
  • Yes. This is what many people overlook - the moment a rebuild starts, the other discs are under stress. Perfect time for them to start showing "issues" ;)
    – TomTom
    Mar 10, 2010 at 19:57
1

The Subject of File Server is deep. However very important aspects are easy to inform about. In the order of importance:

  • For critical data you need server class cpu and ECC ram
  • Against bit rot you need quality drives which has quality firmware. Not necessarily SAS. SATA is ok.
  • You want backward compatibility and openness. Raid chips - raid cards may not be able to reconstruct previous versions raids when they fail (usually their onboard ram wears down). So go for software raid, mdadm is the way.
  • You don't want complexity unless you need something special like snapshots - dedupe, live expandability adding new disks. Then you need to go with LVM , ZFS or such . BTRFS is only reliable with single drive use.
  • You may consider using bcache instead of LVM cache group or volume. To reduce complexity.
  • raid5 or raid6 are problematic. When drives start to fail, recovery becomes nearly impossible. Because slow recovery on failing drives is not practical mostly impossible. Blackblaze reports 4-5 years of heavy duty use for a drive to fail.
  • raid10 is reliable enough. If you go with many small drives less problem will occur recovering, so you may go raid50 or raid60 however small drives are either not top quality or manufacturing dates are old.

To sum it up: For Normal Budget, New Small Case with little Xeon or Epyc inside with ECC ram, mdadm on quality SATA drives (freshly manufactured). Fiber optic cable and 2x 10G Ethernet card utilising ISCSI or SAMBA.

For very high budget, Just buy bigger and better versions in quantity.

For almost no budget, use your existing hardware with small drives in mdadm raid0 with dm-integrity and leave a big external drive for slow over-night backups.

And the most important thing, breaking order of importance just to emphasize at the end, which you can see every time everywhere "RAID is NOT BACKUP" So raid0 makes perfect sense. Raid10 just provides a window to keep working on your time sensitive critical data without requiring emergency recovery procedure to keep on functioning.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .