Using ZFS to take advantage of some of the available options and to manage volumes, not the RAID. I have a single logical device (HW RAID) added to a zpool.
The ZFS ARC does not seem to perform as well as my HW RAID cache so I was trying to disable it to see if I could produce the similar results to the benchmarks run on the HW RAID device, but the performance suffers on the ZFS volumes.
I tried disabling the primarycache and secondarycache but it actually hurt performance, it didn't resolve to using the HW RAID cache as I expected. So I'm at a loss. Is it impossible to use my HW RAID cache with ZFS? Maybe primarycache and secondarycache aren't the right parameters to be modifying.
Configuration HP P410i RAID10 with Writeback BBU Cache.
Zpool with single logical device from the RAID
Created a test sparse zvol for testing device speeds (/dev/zd0)
Update to This Question The lack of performance was caused by ZFS overhead. When ZoL's ARC (primarycache) is disabled, there is extreme overhead at this time, especially on random writes. I'm not sure if this is specific to ZoL or ZFS in general. I recommend at least leaving primarycache=metadata if you are looking to reduce ARC size but maintain performance of your disks.