0

tl;dr - My ZFS RAIDZ2 array reads at 7.5+ GB/s and writes at 2.0+ GB/s when I specify a bs=128K or greater with dd. OS X is assuming 1K (as per stat -f %k .) and all my is ~300MB/s; dd gives the same performance with bs=1k. Even a bs=4k gives 1.1GB/s with dd.

What can I do to improve general I/O to at least 1GB/s?

--

Details:

I am running a 16-drive SATA3 RAIDZ2 OpenZFS on OSX (v1.31r2) filesystem (v5000)over Thunderbolt 2 (twin Areca 8050T2's) to a 12-core 64GB Mac Pro.

The ZFS filesystem was created with ashift=12 (Advanced Format HDD's with 4096 byte blocks) and a recordsize=128k.

I'm seeing transfer rates around 300MB/s from the array in OS X and from the terminal using default commands (note file being copied is 10GB random data):

Regular copy:

Titanic:lb-bu admin$ time cp big-test.data /dev/null

real    0m23.730s
user    0m0.005s
sys     0m12.123s

≈ 424 MB/s

--

dd with bs=1k:

Titanic:lb-bu admin$ time dd if=./big-test.data of=/dev/null bs=1024
9841180+0 records in
9841180+0 records out
10077368320 bytes transferred in 32.572506 secs (309382653 bytes/sec)

real    0m32.575s
user    0m1.880s
sys     0m30.695s

≈ 309 MB/s

--

dd with bs=4k

Titanic:lb-bu admin$ time dd if=./big-test.data of=/dev/null bs=4096
2460295+0 records in
2460295+0 records out
10077368320 bytes transferred in 8.686014 secs (1160183301 bytes/sec)

real    0m8.688s
user    0m0.460s
sys     0m8.228s

≈1.16 GB/s

-- dd with bs=2m:

Titanic:lb-bu admin$ time dd if=./big-test.data of=/dev/null bs=2m
4805+1 records in
4805+1 records out
10077368320 bytes transferred in 1.162891 secs (8665788130 bytes/sec)

real    0m1.165s
user    0m0.003s
sys     0m1.162s

≈8.67 GB/s

--

OS X's read of the boot drive optimal I/O block size (1TB SSD, HFS+):
Titanic:lb-bu admin$ stat -f %k /
4096

--

OS X's read of the array's optimal I/O block size (16-drives RAIDZ2, ZFS):
Titanic:lb-bu admin$ stat -f %k .
1024

--

I also created a ZFS volume on the pool along side the filesystem, and formatted as HFS+. I got the same performance as above.

I'm running ~20-30x below optimal! What am I missing? Any ideas?

--

Update: High speeds were cached I/O (thanks @yoonix). Speeds of ≈300MB/s still seem too slow for this hardware.

@qasdfdsaq: CPU utilization during I/O is negligible (All cores <5%).

zfs get all output:

NAME            PROPERTY               VALUE                    SOURCE
lb-bu           type                   filesystem               -
lb-bu           creation               Tue Sep 30 16:41 2014    -
lb-bu           used                   36.8T                    -
lb-bu           available              10.0T                    -
lb-bu           referenced             138M                     -
lb-bu           compressratio          1.00x                    -
lb-bu           mounted                yes                      -
lb-bu           quota                  none                     default
lb-bu           reservation            none                     default
lb-bu           recordsize             128K                     default
lb-bu           mountpoint             /Volumes/lb-bu           local
lb-bu           sharenfs               off                      default
lb-bu           checksum               on                       default
lb-bu           compression            lz4                      local
lb-bu           atime                  on                       default
lb-bu           devices                on                       default
lb-bu           exec                   on                       default
lb-bu           setuid                 on                       default
lb-bu           readonly               off                      default
lb-bu           zoned                  off                      default
lb-bu           snapdir                hidden                   default
lb-bu           aclmode                discard                  default
lb-bu           aclinherit             restricted               default
lb-bu           canmount               on                       default
lb-bu           xattr                  on                       default
lb-bu           copies                 1                        default
lb-bu           version                5                        -
lb-bu           utf8only               on                       -
lb-bu           normalization          formD                    -
lb-bu           casesensitivity        insensitive              -
lb-bu           vscan                  off                      default
lb-bu           nbmand                 off                      default
lb-bu           sharesmb               off                      default
lb-bu           refquota               none                     default
lb-bu           refreservation         none                     default
lb-bu           primarycache           all                      default
lb-bu           secondarycache         all                      default
lb-bu           usedbysnapshots        0                        -
lb-bu           usedbydataset          138M                     -
lb-bu           usedbychildren         36.8T                    -
lb-bu           usedbyrefreservation   0                        -
lb-bu           logbias                latency                  default
lb-bu           dedup                  off                      default
lb-bu           mlslabel               none                     default
lb-bu           sync                   standard                 default
lb-bu           refcompressratio       1.01x                    -
lb-bu           written                138M                     -
lb-bu           logicalused            36.8T                    -
lb-bu           logicalreferenced      137M                     -
lb-bu           snapdev                hidden                   default
lb-bu           com.apple.browse       on                       default
lb-bu           com.apple.ignoreowner  off                      default
lb-bu           com.apple.mimic_hfs    off                      default
lb-bu           redundant_metadata     all                      default
lb-bu           overlay                off                      default
6
  • 2
    When you read the same 'random' data repeatedly, it's not random. It's cached in RAM. I suspect your first two tests were bottlenecked by the read speed. man 8 purge
    – user143703
    Jun 12, 2015 at 18:59
  • It seems you are correct. I had done several unmounts throughout my testing and thought I was running from uncached data, but I just tried the fastest read with the cache flushed bs=2m and found it to be only slightly faster than "standard" at ≈391 MB/s. Man, I feel dumb for missing the obvious, but thank you. The array should be able to perform much faster so I will keep looking for the cause. Any idea why the array should be so slow, then? <400MB/s?
    – U007D
    Jun 12, 2015 at 20:40
  • Pretty hard to say given the information you've provided so far. You should start off with providing the configuration of your ZFS dataset (zfs get all) and check whether your hard drives or CPU are bottlenecking (iostat/top).
    – noitsbecky
    Jun 12, 2015 at 21:13
  • @qasdfdsaq: updated question with answers to your questions. TIA!
    – U007D
    Jun 13, 2015 at 0:09
  • What about physical I/O? Are you able to read raw data off the drives (e.g. using DD) simultaneously at higher speeds, and does GNU iostat show high disk busy times? My guess is either a physical I/O bottleneck or a single-core CPU bottleneck (4-5% would be 100% of a single core on a 12-core HT system)
    – noitsbecky
    Jun 14, 2015 at 1:54

1 Answer 1

0

You didn't post the zpool status for this but you imply in the post that all 16 disks are in a single vdev in RAIDZ2. While this is a good, safe configuration you have to understand that RAIDZ isn't designed primarily for speed. It is designed to be near bulletproof. RAIDZ2 is analogous to RAID6 but the variant has features that make it slower and safer.

See this nice write up for the full details but these two quotes should help you see the issue (emphasis mine):

When writing to RAID-Z vdevs, each filesystem block is split up into its own stripe across (potentially) all devices of the RAID-Z vdev. This means that each write I/O will have to wait until all disks in the RAID-Z vdev are finished writing. Therefore, from the point of view of a single application waiting for its IO to complete, you'll get the IOPS write performance of the slowest disk in the RAID-Z vdev.

When reading from RAID-Z vdevs, the same rules apply, as the process is essentially reversed (no round robin shortcut like in the mirroring case): Better bandwidth if you're lucky (and read the same way as you've written) and a single disk's IOPS read performance in the majority of cases that matter.

In effect, you have 16 medium speed drives and for each write pass, you wait for all 16 drives to check in with the controller and say "done" before the next write starts. With 16 disks, you are effectively always going to wait for a near-full disk rotation before one of the writes so you are held up by physics and how ZFS commits data.

Single process/thread write tasking isn't the best case for ZFS in general. Running multiple small data read/write tasks at once might show you better IOPS numbers but I think the physics of ZFS is your main issue.

If you are willing to sacrifice the space, mirroring would be likely be faster. You might also get marginally better performance out of ZFS by creating 2 8-disk RAIDZ2 vdevs in the pool rather than 1 16-disk RAIDZ2 vdev. That, too, will cost you usable storage space but may help the commits happen faster.

Unfortunately I don't have good news for you.

1
  • I've seen at least one person reporting 6-10x read performance of what I'm getting under RAID-Z2 (2.5GB). For streaming reads, I'd have expected significantly higher figures, but you may be right. Thank you for taking a moment to share your thoughts.
    – U007D
    Jun 25, 2015 at 16:18

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .