I am getting poor write performance on a NFS share hosted with OpenIndiana 151a. I have 2x 1 TB Mirror (Seagate constellation drives), 2 SSD drives, 1 read cache and 1 write cache. My thinking was, that by using a write cache I would gain SSD like performance even though I have a pretty slow hard disk. I am getting an avg of 40 MB/s. I should note that both my SSD drives are SATA3 capable of 500 MB/s. I feel cheated! My setup is as follows:

  • ESXi 5, NFS datastore, MTU set to 9000 on VMkernel and vSwitch
  • VM1 OpenIndiana hosting the datastore, resides on direct attached storage.
  • VM2 Windows XP with 2 10GB hard drive, 1 system drive, 2nd testing drive, using HD tune pro, resides on NFS datastore.

enter image description here enter image description here enter image description here enter image description here

I am busy running a spectrum of IOmeter tests, will post results when I am done.

I am not seeing the advertised SATA3 speeds here, would I just be better off using the SSD's as direct attached storage? In other words, is NFS the problem here?

  • Please provide more information on how you're testing the speed. Are you using two vSwitches? – ewwhite Mar 14 '12 at 15:51
  • Nope just 1 vSwitch with both the OpenIndiana (NAS) VM connected, and the Windows XP test VM connected, the VMkernel is also on the switch. The whole test vm as well as the 10Gb virtual disk pictured above is stored on the NFS share that is provided through OI. – cmaduro Mar 14 '12 at 18:36
  • You should separate the storage traffic onto another switch. One interface from the OpenIndiana box should be serving ESXi on a dedicated switch. See my answer below. – ewwhite Mar 14 '12 at 18:41
  • 1
    Please please please, update your question and title to use MB/GB where appropriate. These are very different than Mb/Gb by a factor of 8. – MDMarra Mar 14 '12 at 18:54

This depends entirely on what and how you're testing. Is that 40Mb/sec read or write speed? Or combined? If your benchmarking software is all sequential reads or writes, chances are that you won't see a benefit from the SSD cache units.

Your virtual switch setup should look something like this. I'm using NexentaStor instead of OpenIndiana, but the same principles apply. The storage server should have a private switch with a VMkernel port. That's to provide NFS to the ESXi host. Then, you present a datastore to the ESXi system and place your VMs on it. You don't need a physical adapter, and using the VMXNET3 network adapter, things should show up as 10GbE.

enter image description here

  • I see the VMKernel port seems to be on another network, or is your subnet mask here – cmaduro Mar 14 '12 at 20:30
  • I have two VMkernel ports in this example. Mask is /24. The vSwitch1 is for communication between ESXi and the Nexenta NFS datastore. – ewwhite Mar 14 '12 at 20:36
  • What would be the ip of the datastore (nexenta box) then in this example? Im talking about vSwitch1. – cmaduro Mar 14 '12 at 20:51
  • I would use The Nexenta server has two interfaces. One for communication with ESXi, the other for management and ssh access (192.168.10.x). – ewwhite Mar 14 '12 at 21:01
  • I ask because I cannot connect to the datastore when I setup OI and the VMkernel on a seperate vswitch. I am using subnet for this switch. I point to the OI box ( when adding the datastore. My VMkernel's IP is set to, and none of the VMkernel port options are selected. What am I doing wrong? – cmaduro Mar 14 '12 at 21:03

As far as I know ZIL isn't a "write cache", more a performant journal, thus you'd still be getting the raw disk's speed at best. Although 40MB/s is less than that and that's where "how did you do those tests" comes into play.

Correction: I was told I'm wrong and ZIL does double function as write cache, but not sure how/if this is all correct.

  • All (synchronous) writes are written to ARC (in mem), and if slog is configured then (synchronous) writes are written to it, and when successful: acknowledged back to client. Flushing is performed (about) once per 5 seconds iirc. In some sense this also reduce the random writes back to the disks, as the data to be flushed can be written in an as organized manner as possible (in terms of head movement). Needless to say; if no slog/zil, written to disks (or bbu), then acknowledged. – 3molo May 30 '12 at 19:20
  • Btw, you are probably correct. For a continuous stream of written (synchronous) data, it will be limited to the write speed of the (given i/o pattern of the) disks. It's really more useful to achieve high random iops than to raw synchronous continuous writes. – 3molo May 30 '12 at 19:26
  • thanks 3molo, this raises the level of detail to a useful amount of information!! – Florian Heigl May 30 '12 at 22:35
  • Very welcome! I say probably because I am still in my infancy regarding ZFS. If I am not correct, someone please correct me. – 3molo May 31 '12 at 6:44

vSphere mounts all NFS shares with sync, meaning all writes are done synchronous. If you instead mount the LUNs over iSCSI - all specifically synchronous writes will be synchronous - and the rest will be asynchronous.

To answer regarding the SSD as ZIL then; For a continuous stream of written (synchronous) data, it will be limited to the write speed of the (given i/o pattern of the) disks. ZIL is really more useful to achieve high random iops than to raw synchronous continuous writes.

In most cases, the amount of sequential data is not as important of a factor as how many truly random writes and reads you can handle - especially not in virtualized environments where you are likely to have many different "clients" hitting the storage. Consider async writes if you need high throughput by using iSCSI instead of NFS.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.