I'm running a large ZFS pool built for 256K+ request size sequential reads and writes via iSCSI (for backups) on Ubuntu 18.04. Given the need for high throughput and space efficiency, and less need for random small-block performance, I went with striped raidz2 over striped mirrors.
However, the 256K sequential read performance is far lower than I would have expected (100 - 200MBps, peaks up to 600MBps). When the zvols are hitting ~99% iowait in iostat, the backing devices typically run between 10 and 40% iowait, which suggests to me the bottleneck is something I'm missing in configuration, given it shouldn't be the backplane or CPUs in this system, and sequential workloads shouldn't work the ARC too hard.
I've played quite a bit with module parameters (current config below), read hundreds of articles, issues on OpenZFS github, etc. Tuning prefetch and aggregation got me to this performance level - by default, I was running at about ~50MBps on sequential reads as ZFS was sending TINY requests to the disks (~16K). With aggregation and prefetch working OK (I think), disk reads are much higher, around ~64K on average in iostat.
NICs are LIO iscsi target with cxgbit offload + Windows Chelsio iscsi initiator work well outside the ZFS zvols, with a optane directly mapped returning nearly full line-rate on the NICs (~3.5GBps read and write).
Am I expecting too much? I know ZFS prioritizes safety over performance, but I'd expect a 7x9 raidz2 to provide better sequential reads than a single 9-drive mdadm raid6.
System specs and logs / config files:
Chassis: Supermicro 6047R-E1R72L HBAs: 3x 2308 IT mode (24x 6Gbps SAS channels to backplanes) CPU: 2x E5-2667v2 (8 cores @ 3.3Ghz base each) RAM: 128GB, 104GB dedicated to ARC HDDs: 65x HGST 10TB HC510 SAS (9x 7-wide raidz2 + 2 spares) SSDs: 2x Intel Optane 900P (partitioned for mirrored special and log vdevs) NIC: Chelsio 40GBps (same as on initiator, both using hw offloaded iSCSI) OS: Ubuntu 18.04 LTS (using latest non-HWE kernel that allows ZFS SIMD) ZFS: 0.8.1 via PPA Initiator: Chelsio iSCSI initiator on Windows Server 2019
ashift=12 recordsize=128K (blocks on zvols are 64K, below) compression=lz4 xattr=sa redundant_metadata=most atime=off primarycache=all
sparse volblocksize=64K (matches OS allocation unit on top of iSCSI)
7x 9-wide raidz2 mirrored 200GB optane special vdev (SPA metadata allocation classes) mirrored 50GB optane log vdev
# 52 - 104GB ARC, this system does nothing else options zfs zfs_arc_min=55834574848 options zfs zfs_arc_max=111669149696 # allow for more dirty async data options zfs zfs_dirty_data_max_percent=25 options zfs zfs_dirty_data_max=34359738368 # txg timeout given we have plenty of Optane ZIL options zfs zfs_txg_timeout=5 # tune prefetch (have played with this 1000x different ways, no major improvement except max_streams to 2048, which helped, I think) options zfs zfs_prefetch_disable=0 options zfs zfetch_max_distance=134217728 options zfs zfetch_max_streams=2048 options zfs zfetch_min_sec_reap=3 options zfs zfs_arc_min_prefetch_ms=250 options zfs zfs_arc_min_prescient_prefetch_ms=250 options zfs zfetch_array_rd_sz=16777216 # tune coalescing (same-ish, increasing the read gap limit helped throughput in conjunction with low async read max_active, as it caused much bigger reads to be sent to the backing devices) options zfs zfs_vdev_aggregation_limit=16777216 options zfs zfs_vdev_read_gap_limit=1048576 options zfs zfs_vdev_write_gap_limit=262144 # ZIO scheduler in priority order options zfs zfs_vdev_sync_read_min_active=1 options zfs zfs_vdev_sync_read_max_active=10 options zfs zfs_vdev_sync_write_min_active=1 options zfs zfs_vdev_sync_write_max_active=10 options zfs zfs_vdev_async_read_min_active=1 options zfs zfs_vdev_async_read_max_active=2 options zfs zfs_vdev_async_write_min_active=1 options zfs zfs_vdev_async_write_max_active=4 # zvol threads options zfs zvol_threads=32
I'm tearing my hair out on this. Pressure's on from users to go all-Windows with Storage Spaces, but I've used parity storage spaces (even with Storage Spaces Direct with mirrors on top), and it's not pretty either. I'm tempted to go straight mdadm raid60 under iSCSI, but would love it if someone could point out something boneheaded I'm missing that will unlock performance with the bitrot protection of ZFS :)