8

Which is best ?

Mirroring 2 internal disks with HW raid and then putting ZFS filesystem on top

or

Using the 2 internal disks without HW Raid and creating a ZFS mirror with the 2 disks.

I've read that to get the full benifits of ZFS it needs at least 2 disks to use its error correction and prediction features fully.

Its a question of ZFS ontop of HW RAID , or just ZFS , not HW vs ZFS . Performance is not my issue.

2
  • What kind of hardware RAID? Aug 11, 2011 at 15:58
  • It was the internal RAID card on an Oracle X4470. RAID 1 with the two internal disks
    – AndyM
    Sep 9, 2011 at 12:12

4 Answers 4

13

ZFS works optimally if it knows as much as possible about the underlying physical disk structure. A hardware RAID 1 will present a single volume to the OS which ZFS will then treat as a single disk. You'll therefore miss out on the benefits of error correction / fault detection / checksumming etc.

There's another benefit to using ZFS over HW RAID - if your hardware dies, be it the motherboard, RAID card, CPU etc, you can pull the disks and use them in any other system that is running ZFS, and the entire filesystem is readable. If you use HW RAID, you may have a dependency on that specific RAID card, so if it fails you might need to find a similar model before you can read the disks again. You can mitigate this by having dual RAID cards, but this usually starts to become expensive and complex.

In my experience, for the majority of use-cases that I've used a mirrored disk configuration, the performance difference between ZFS and HW RAID wasn't great. If performance is your #1 priority, HW RAID will offer the best solution. In all other cases, I believe ZFS is the better choice.

17
  • How can having to write data twice immediately over the bus come close to the performance of writing once immediately to RAM, regardless of how much RAM and CPU you've got?
    – womble
    Aug 11, 2011 at 8:59
  • Not sure what you mean - both HW RAID and ZFS will do write caching...? In fact, ZFS lets you specify an entire set of disks as write cache devices if you want to.
    – growse
    Aug 11, 2011 at 9:06
  • 1
    If write latency is that important to you, then use HW RAID. In all other cases, I believe that the advantages of ZFS outweigh those of HW RAID. YMMV. Obviously.
    – growse
    Aug 11, 2011 at 9:22
  • 4
    @AndyM: If you want availability then ZFS is a better option, hardly any HW RAID solution checks both mirrors when reading let alone adds checksums to written data. Aug 11, 2011 at 11:12
  • 2
    @growse: You no longer HAVE to mirror the ZIL disks. In the more recent Zpool vers (26+ IIRC) you can safely remove a ZIL disk from a pool. The pool simply reverts to using the pool disks itself for the ZIL. If you're doing a lot of sync writes then you'd take a performance hit until you replace the ZIL disk.
    – Tim
    Aug 11, 2011 at 12:42
8

Although I love ZFS if you're doing a LOT of writing then take into consideration that any software RAID 1/10 will need to send every write twice over the bus and through the controller - meaning you can get into congestion and queue issues.

2
  • This is a good point - from a performance point of view, HW RAID has some significant advantages in certain situations.
    – growse
    Aug 11, 2011 at 9:14
  • 2
    On the other hand, if you have close to 150MiB/s random writes (it's a db server) you're a madman for running it on a single bus. Interface bandwidth is hardly a problem. Aug 11, 2011 at 11:15
2

ZFS has lots of built in optimizations that work best when used without Raid controllers. On a ZFS Raid1 you get read performance that is usually superior to a hardware Raid, also with RaidZ you can get superior performance to Hardware Raid5 Just take a look at Anadtech.

What you absolutely need to take into consideration is how ZFS caches though. You should basically have 4GB of Ram to use it at all, 8GB to get some decent performance, but more is always good. There are lots of cases where more Ram helps with ZFS. The best ways to improve performance though are seperate devices (SSDs) for the ZIL and L2ARC, as those generally tend to be bottlenecking performance. This explains the idea behind L2ARC a little.

The principal problem with Hardware Raid here is, that ZFS does, what the Raid Controller does, so when using a Hardware Raid below ZFS, there is a Raid Controller that expects itself and only itself to be doing Queueing, Caching, IO Optimization and so on and there is ZFS on top of it also expecting the same of itself. This has side effects of course and they aren't necessarily pretty and will most likely have negative performance effects due to the way ZFS does I/O Parallelization. Also its predictive "self healing" features will in part be kind of useless that way. Therefor it is specifically recommended that when you use a Raid controller, you should expose the Disks JBOD are as Single-Disk Raid0 devices, so that ZFS can do the housekeeping by itself.

-1

Well, you need to define "best". I'd expect that a good hardware RAID controller with BBU would provide superior performance, but perhaps performance isn't your primary consideration.

2
  • And my answer is ZFS ontop of HW RAID or just ZFS.
    – womble
    Aug 11, 2011 at 10:08
  • 1
    That is actually a very bad idea. It will basically not be good for ZFSs I/O Parallelization and for exactly this reason its probably not good for the disks either.
    – juwi
    Jan 26, 2012 at 1:07

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .