I am currently testing ZFS (Opensolaris 2009.06) in an older fileserver to evaluate its use for our needs. Our current setup is as follows:

  • Dual core (2,4 GHz) with 4 GB RAM
  • 3x SATA controller with 11 HDDs (250 GB) and one SSD (OCZ Vertex 2 100 GB)

We want to evaluate the use of a L2ARC, so the current ZPOOL is:

$ zpool status
  pool: tank
 state: ONLINE
 scrub: none requested

        NAME         STATE     READ WRITE CKSUM
        afstank      ONLINE       0     0     0
          raidz1     ONLINE       0     0     0
            c11t0d0  ONLINE       0     0     0
            c11t1d0  ONLINE       0     0     0
            c11t2d0  ONLINE       0     0     0
            c11t3d0  ONLINE       0     0     0
          raidz1     ONLINE       0     0     0
            c13t0d0  ONLINE       0     0     0
            c13t1d0  ONLINE       0     0     0
            c13t2d0  ONLINE       0     0     0
            c13t3d0  ONLINE       0     0     0
          c14t3d0    ONLINE       0     0     0

where c14t3d0 is the SSD (of course). We run IO tests with bonnie++ 1.03d, size is set to 200 GB (-s 200g) so that the test sample will never be completely in ARC/L2ARC. The results without SSD are (average values over several runs which show no differences)

write_chr       write_blk       rewrite         read_chr        read_blk        random seeks
101.998 kB/s    214.258 kB/s    96.673 kB/s     77.702 kB/s     254.695 kB/s    900 /s

With SSD it becomes interesting. My assumption was that the results should be in worst case at least the same. While write/read/rewrite rates are not different, the random seek rate differs significantly between individual bonnie++ runs (between 188 /s and 1333 /s so far), average is 548 +- 200 /s, so below the value w/o SSD.

So, my questions are mainly:

  1. Why do the random seek rates differ so much? If the seeks are really random, they should not differ much (my assumption). So, even if the SSD is impairing the performance it should be the same in each bonnie++ run.
  2. Why is the random seek performance worse in most of the bonnie++ runs? I would assume that some part of the bonnie++ data is in the L2ARC and random seeks on this data performs better while random seeks on other data just performs similarly like before.
  • When using L2ARC, don't you need two drives in RAID1? I don't know if this should affect your performance since I don't know ZFS to that extent. But, I do know that when using RAID1 in ZFS, it is smart enough to read chunk X from one drive and chunk Y from another drive, to even the load and getting the best throughput. – tore- Aug 4 '10 at 12:41
  • Ok, I don't know what you mean in detail. A RAID1 of SSDs for the L2ARC or one RAID1 for the data disks? Also I'm not saying this is an ideal setup. Of course, ZFS can do some smart I/O handling to get a good performance out of the two RAIDZs. The point which baffles me though is: ZFS is able to distribute random reads quite good so that the random read performance is okay. An additional SSD could perhaps increase this performance as L2ARC (low latency for random accesses and so on), but it should not impair it that much. – Florian Kruse Aug 4 '10 at 13:20
  • 1
    You can watch the disk activity with iostat -XCn 1 and see if the SSD's asvc_t is too high for some reason. It should usually be fairly low (<1ms). – Giovanni Tirloni Aug 8 '10 at 0:15

Not sure why you're seeing the behavior you're seeing, but I can tell you why they don't necessarily signal terrible real world ZFS performance. Bonnie is designed to measure performance of the actual disks and intentionally tries to not leverage disk/memory cache. You're trying to use it to measure disk cache.

It can take hours for an L2ARC device to get hot. Depending on how much main memory you have for ARC and the disk performance with your workload, a 100GB SSD for L2ARC will take 1-2hrs to get hot and perhaps even longer (Source). Secondly L2Arc is designed to cache random reads and not streaming read workloads. "If you use the L2ARC for a streaming or sequential workload, then the L2ARC will mostly ignore it and not cache it" (Source). I would be really surprised if much of your bonnie workload ever made it into L2ARC. Instead of using bonnie++, try to generate load which resembles your real use of the system.

Although unlikely, running the latest development versions of Bonnie++ (1.96 vs 1.03d) may produce something closer to your expected results. You may also want to check out this sun blogs post about bonnie++ with the Sun 7000 Series, though they run over NFS and not locally.

  • What you say corresponds to my further observations. Testing random seeks with bonnie++ 1.96, the results do not vary significantly between runs anymore and are in average slightly higher than runs without L2ARC. Warmup is an issue here. A custom random seek benchmark tool I wrote shows a very satisfying random seek improvement over time. Our real use of the system is a file server for home directories/software, so it's not exactly easy to design a benchmark for that, but we expect lots of random access to the data. That's why I was specifically interested in L2ARC. – Florian Kruse Aug 12 '10 at 6:42

You can also enable streaming data cache in the L2ARC in your SSD.

The l2arc_noprefetch parameter is useful for that.

This tunable determines whether streaming data is cached or not. The default is not to cache streaming data (the default value is true). If you set it to false, you will probably see your L2ARC cache hit ratio increase significantly.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.