Will adding a SSD cache device to my ZFS storage improve performance?

Question

The server has 4GB of RAM and my zpool is made up of 15.5k SAS drives arranged like this:

    NAME          STATE     READ WRITE CKSUM
    tank          ONLINE       0     0     0
      raidz1-0    ONLINE       0     0     0
        c0t2d0    ONLINE       0     0     0
        c0t3d0    ONLINE       0     0     0
        c0t4d0    ONLINE       0     0     0
        c0t5d0    ONLINE       0     0     0
        c0t6d0    ONLINE       0     0     0
        c0t7d0    ONLINE       0     0     0
        c0t8d0    ONLINE       0     0     0
      raidz1-1    ONLINE       0     0     0
        c0t10d0   ONLINE       0     0     0
        c0t11d0   ONLINE       0     0     0
        c0t12d0   ONLINE       0     0     0
        c0t13d0   ONLINE       0     0     0
        c0t14d0   ONLINE       0     0     0
    spares
      c0t9d0      AVAIL
      c0t1d0      AVAIL   

The primary use is as an NFS store for a couple VMWare ESXi servers. I can't do any "true" benchmarks because this is a production system (no budget for test systems), but using dd and bonnie++ I can't get more than ~40-50MB/s writes and ~70-90MB/s reads. It seems I should be able to do much better, but I'm not sure where to optimize. Based on what I've read, I think dropping in a OCZ Vertex 2 Pro SSD as my L2ARC is going to be the best bang-for-the-buck to improve througput.

1. Is there something else I should be looking into to help performance? If not...
2. How do I know how big a cache device I need?
3. Am I safe with only a single SSD as my cache device?

Answer 1

My guess is that the ZFS Intent Log (ZIL) is limiting the write performance. A fast SSD used as a log device could help. Some hints:

• ZIL Accelerator: DRAM or Flash? explains the issues in detail.
• You may want to test with a disabled ZIL. See my other posting.
• Checkout ZFS Features in Oracle Solaris Express if you're using a recent ZFS release because the way how to disable the ZIL has changed. There's a new sync filesystem property now.
• Until recently it was impossible to remove a log device from a ZFS pool. Of course, this caused problems if the single log devices failed. It was therefore recommended to use a mirror for the log device. However, recent ZFS releases fixed this limitation and it's now possible to remove the single log device with zpool remove. Make sure you use a version that supports this feature if you really going to install the SSD.

Answer 2

I definitely would do it.. I have read that it helps substantially.. However you will need to get a quality SSD, as it will see A LOT of use, so the wear leveling should function well..

I found this today.. which would be very nice

http://www.engadget.com/2010/11/08/ocz-revodrive-x2-pcie-ssd-reviewed-blisteringly-fast-in-every-c/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed:+weblogsinc/engadget+(Engadget)

1. not sure
2. 30 times your RAM should be enough.. ( 300GB plus should help buffer)
3. The single SSD should be enough,its only a cache remember for throughput.. (Things will just get slow when it dies/breaks/doesnt exist)