tl;dr - My ZFS RAIDZ2 array reads at 7.5+ GB/s and writes at 2.0+ GB/s when I specify a bs=128K
or greater with dd
. OS X is assuming 1K (as per stat -f %k .
) and all my is ~300MB/s; dd
gives the same performance with bs=1k
. Even a bs=4k
gives 1.1GB/s with dd.
What can I do to improve general I/O to at least 1GB/s?
--
Details:
I am running a 16-drive SATA3 RAIDZ2 OpenZFS on OSX (v1.31r2) filesystem (v5000)over Thunderbolt 2 (twin Areca 8050T2's) to a 12-core 64GB Mac Pro.
The ZFS filesystem was created with ashift=12
(Advanced Format HDD's with 4096 byte blocks) and a recordsize=128k
.
I'm seeing transfer rates around 300MB/s from the array in OS X and from the terminal using default commands (note file being copied is 10GB random data):
Regular copy:
Titanic:lb-bu admin$ time cp big-test.data /dev/null
real 0m23.730s
user 0m0.005s
sys 0m12.123s
≈ 424 MB/s
--
dd
with bs=1k
:
Titanic:lb-bu admin$ time dd if=./big-test.data of=/dev/null bs=1024
9841180+0 records in
9841180+0 records out
10077368320 bytes transferred in 32.572506 secs (309382653 bytes/sec)
real 0m32.575s
user 0m1.880s
sys 0m30.695s
≈ 309 MB/s
--
dd
with bs=4k
Titanic:lb-bu admin$ time dd if=./big-test.data of=/dev/null bs=4096
2460295+0 records in
2460295+0 records out
10077368320 bytes transferred in 8.686014 secs (1160183301 bytes/sec)
real 0m8.688s
user 0m0.460s
sys 0m8.228s
≈1.16 GB/s
--
dd
with bs=2m
:
Titanic:lb-bu admin$ time dd if=./big-test.data of=/dev/null bs=2m
4805+1 records in
4805+1 records out
10077368320 bytes transferred in 1.162891 secs (8665788130 bytes/sec)
real 0m1.165s
user 0m0.003s
sys 0m1.162s
≈8.67 GB/s
--
OS X's read of the boot drive optimal I/O block size (1TB SSD, HFS+):
Titanic:lb-bu admin$ stat -f %k /
4096
--
OS X's read of the array's optimal I/O block size (16-drives RAIDZ2, ZFS):
Titanic:lb-bu admin$ stat -f %k .
1024
--
I also created a ZFS volume on the pool along side the filesystem, and formatted as HFS+. I got the same performance as above.
I'm running ~20-30x below optimal! What am I missing? Any ideas?
--
Update: High speeds were cached I/O (thanks @yoonix). Speeds of ≈300MB/s still seem too slow for this hardware.
@qasdfdsaq: CPU utilization during I/O is negligible (All cores <5%).
zfs get all output:
NAME PROPERTY VALUE SOURCE
lb-bu type filesystem -
lb-bu creation Tue Sep 30 16:41 2014 -
lb-bu used 36.8T -
lb-bu available 10.0T -
lb-bu referenced 138M -
lb-bu compressratio 1.00x -
lb-bu mounted yes -
lb-bu quota none default
lb-bu reservation none default
lb-bu recordsize 128K default
lb-bu mountpoint /Volumes/lb-bu local
lb-bu sharenfs off default
lb-bu checksum on default
lb-bu compression lz4 local
lb-bu atime on default
lb-bu devices on default
lb-bu exec on default
lb-bu setuid on default
lb-bu readonly off default
lb-bu zoned off default
lb-bu snapdir hidden default
lb-bu aclmode discard default
lb-bu aclinherit restricted default
lb-bu canmount on default
lb-bu xattr on default
lb-bu copies 1 default
lb-bu version 5 -
lb-bu utf8only on -
lb-bu normalization formD -
lb-bu casesensitivity insensitive -
lb-bu vscan off default
lb-bu nbmand off default
lb-bu sharesmb off default
lb-bu refquota none default
lb-bu refreservation none default
lb-bu primarycache all default
lb-bu secondarycache all default
lb-bu usedbysnapshots 0 -
lb-bu usedbydataset 138M -
lb-bu usedbychildren 36.8T -
lb-bu usedbyrefreservation 0 -
lb-bu logbias latency default
lb-bu dedup off default
lb-bu mlslabel none default
lb-bu sync standard default
lb-bu refcompressratio 1.01x -
lb-bu written 138M -
lb-bu logicalused 36.8T -
lb-bu logicalreferenced 137M -
lb-bu snapdev hidden default
lb-bu com.apple.browse on default
lb-bu com.apple.ignoreowner off default
lb-bu com.apple.mimic_hfs off default
lb-bu redundant_metadata all default
lb-bu overlay off default
man 8 purge
bs=2m
and found it to be only slightly faster than "standard" at ≈391 MB/s. Man, I feel dumb for missing the obvious, but thank you. The array should be able to perform much faster so I will keep looking for the cause. Any idea why the array should be so slow, then? <400MB/s?