Why am I getting only 1gbps between Solaris and VMware?

Question

I have a VMWare ESXi 5.1.0 (build 1065491) host running a Solaris (OpenIndiana) VM as a guest. I have created a private vSwitch for NFS traffic between ESXi and Solaris. The Solaris VM has the VMware tools installed and has a VMXnet3 adaptor (vmxnet3s0) on the private vSwitch.

Reading from a file directly on the Solaris VM using dd I get speeds of up to 4.5 GB/sec (44.8 gigabit/sec) when reading a file (if it has been cached by my ARC/L2ARC)

Reading the same file over NFS from VMware, I can only get a maximum of 0.163 GB/sec (1.131 gigabit/sec)

Why is there such a difference? I expect some overhead due to TCP and NFS, as well as virtual networking, but that difference isn't what I'd call overhead, it looks like something's wrong.

The CPU on the Solaris VM is never more than 20% used. Load averages are low. Plenty of free RAM on the host (it's not swapping) and CPU utilization on the host is well below 50% (typically around 33%). On the guest VM, dladm show-linkprop vmxnet3s0 shows that the speed is 10000 (10 Gbit/sec) and the MTU is 9000 (Jumbo Frames enabled)

As a test I tried using a "standard" E1000 vmnic (e1000g0) and I was able to enable jumbo frames (9000 MTU) however reading the same file over NFS from VMware, I can only get a maximum of 0.124 GB/sec (0.992 gigabit/sec) which makes sense as the link speed on the E1000 is only 1000 megabit.

Why am I only seeing only about 1 gigabit/second transfer speeds between VMware ESXi 5.1 and my OpenIndiana Solaris VM?

---

UPDATE: After applying the performance tuning recommended by VFrontDe in his answer, I'm now getting an impressive 8-9 Gbit/sec between Solaris and a Linux VM with a vmxnet3 adapter. I'm getting ~6 Gbps reading a cached file via NFS under Linux:

root@iperf:/mnt/public/ISO# dd if=Test.cdr of=/dev/null bs=1024k
5884+1 records in
5884+1 records out
6170824704 bytes (6.2 GB) copied, 8.44985 s, 730 MB/s

However Reading that same file on the VMware ESXi host, I get an abysmal 1.91 Gbps:

/vmfs/volumes/3d77318f-82c7901a/ISO # time dd if=Windows\ 7\ Home\ Premium.cdr of=/dev/null bs=1024k
5884+1 records in
5884+1 records out
real    0m 27.85s
user    0m 17.32s
sys     0m 0.00s

6.2 GB in 27.85 seconds works out to be 1.91 gbps.

What is causing Vmware ESXi's vmknic to perform so poorly when vmxnet3 to vmxnet3 is so much better?

Answer 1

It looks like there are some issues and bugs with the vmxnet3 adapter for Solaris. Please have a look at this very detailed post: http://blog.cyberexplorer.me/2013/03/improving-vm-to-vm-network-throughput.html The author was able to get much better throughput after some tweaking, but it's too much to repeat all steps here.