4

I have the following setup for testing: HP Microserver N36L (Dual-core low-power Athlon64) 8GB ECC RAM 2-port Intel Gigabit NIC 4x3TB WD Green (5400 RPM?) 1 sata disk for OS image (FreeBSD 10.1)

Switch is an HP v1910-24G (Managed L2 switch, 24 gigabit ports)

Client is: Core i5 2400 on DH67BL motherboard 24GB RAM 1 Intel Gigabit (onboard)

The FreeBSD machine exposes its zvols over iSCSI and the initiator is Windows. On Windows, the iSCSI volumes are formatted to NTFS, and VMWare Workstation uses it as disk for other VMs (in this case, a Windows 7 VM too).

I've tried several configurations for zvols, mainly volblocksizes of varying values, from 4k to 64k, and NTFS cluster sizes of 4 to 64k as well. I've tried combinations of blocksize=cluster size, and block size > cluster size.

In all cases I get basically the same:

Sequential reads are at 110-120MB/s (Gigabit Ethernet limit). Random reads are <5MB/s

Copying large files outside or inside the VMs is acceptable. Booting windows is painfully slow (>3 minutes on a fresh install to login screen. 1 minute more to desktop)

Is this expected behavior?

2 Answers 2

3

This configuration has a lot of components and design decisions that contribute to its poor performance.

  • The HP Microserver is low-performance item. RAM-constrained and low-end CPU.

  • Western Digital Green drives are low-RPM and require special consideration for their 4k-sector formatting. This is likely the main problem.

  • It seems like you may be using RAIDZ1 across three disks. Random read/write performance under that setup is pretty low as well. Expect to see the performance of a single disk in both cases.

  • No write caching/ZIL device.

  • 1GbE iSCSI to a Windows software initiator isn't great.

  • VMware Workstation definitely doesn't help.

  • Certain ZFS attributes could work against you here: dedupe is bad. gzip compression is bad (lz4 is okay). Too much RAM allocated to ARC is bad.

2
  • 2
    1. The HP Microserver may be low-performance. but local reads (sequential) are 500MB/s according to bonnie++. 2. the pool was created with the mentioned trickery to make it ashift=12. 3. it's raidz1 across 4 disks, but in my case the problem is random read performance 4. 120MB/s to 5MB/s .. i can't believe that's due to windows' iSCSI. The server's CPU is mostly idle as well (10% utilization) 5. there's also poor performance outside vmware 6. dedup is off (was never on), compression is off. arc_max_limit is 6GB
    – hjf
    Feb 15, 2015 at 17:33
  • Well, this doesn't sound like a good setup. Try it with ZFS mirrors instead of RAIDZ.
    – ewwhite
    Feb 15, 2015 at 22:35
2

Mechanical HDDs struggle at random I/O access.

4x WD Green that give you 5 MB/s of random read performance is even better than I expected.
Some math: at 5400 RPM, a single rotation need 11 ms, so an half-rotation need ~5.5ms. This is the rotational delay. Adding another ~8ms of seek time (the time the actuator needs to correctly position the read/write heads) bring total time to ~13ms. This means that IOPS per single disk is 1000ms (1 sec) / 13ms = ~76 IOPS. At 4K random read, this translate in ~300 KB/s

Considering how ZFS RAIDZ itself are quite bad in terms of IOPS (its by design: ZFS sacrifices IOPS for data safety, read here for more details) I am surprised that you have 5 MB/s of random read performance. Did you test with 4KB random reads or bigger ones (eg: 64KB)?

Back to you original problem: I think it is a combination of small iSCSI volblocksize/recordsize, RAIDZ vols and 4K disks. In RAIDZ Vols, each read/write is broken up by the number of data disks and then forwarded to the physical disks. For example on your setup, a single 8K write will be broken in multiple ~3KB writes (3x 3KB writes for data and 1x 3KB write for parity) and this will trigger a read-modify-write on the disks themselves (as they are 4K based disks). As Windows startup is random-write heavy, it can be affected by your disks behavior.

Try this: if you can afford to destroy your ZPOOL/ZDEV, destroy it and recreate it with volblocksize of 16K or 32K (see here). Then reinstall a Windows machine and re-try your scenario.

4
  • 1
    Keep in mind that the recommended ZFS zvol block size for Linux is 128k, whereas it's 8k-32k under Solaris and variants. I have no idea what FreeBSD works best with.
    – ewwhite
    Feb 17, 2015 at 15:06
  • While this is true for local zvol Access, when exporting zvols via iSCSI there are other factors (mainly network related) to take into account. For this reason, zfs default recordsize for iSCSI exported volumes is 8KB. This does not means that you can't use larger block size, but you should benchmark your setup to ensure that they perform well in your network environment.
    – shodanshok
    Feb 17, 2015 at 18:23
  • 1
    Don't you mean "volblocksize"? There's no recordsize property for zvols. As stated in the question, i've tried different combinations of volblocksize and NTFS cluster size. In all cases random access is terrible.
    – hjf
    Feb 17, 2015 at 19:29
  • Yes, as you are exporting the "raw" vdev, volblocksize is the right propriety. I edited the answer to clarify that. Anyway, with slow HDD random read/write performance will struggle. Can you run one of the suggested benchmark?
    – shodanshok
    Feb 17, 2015 at 20:16

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .