I have a VMWare ESXi 5.1.0 (build 1065491) host running a Solaris (OpenIndiana) VM as a guest. I have created a private vSwitch for NFS traffic between ESXi and Solaris. The Solaris VM has the VMware tools installed and has a VMXnet3 adaptor (
vmxnet3s0) on the private vSwitch.
Reading from a file directly on the Solaris VM using
dd I get speeds of up to 4.5 GB/sec (44.8 gigabit/sec) when reading a file (if it has been cached by my ARC/L2ARC)
Reading the same file over NFS from VMware, I can only get a maximum of 0.163 GB/sec (1.131 gigabit/sec)
Why is there such a difference? I expect some overhead due to TCP and NFS, as well as virtual networking, but that difference isn't what I'd call overhead, it looks like something's wrong.
The CPU on the Solaris VM is never more than 20% used. Load averages are low. Plenty of free RAM on the host (it's not swapping) and CPU utilization on the host is well below 50% (typically around 33%). On the guest VM,
dladm show-linkprop vmxnet3s0 shows that the speed is 10000 (10 Gbit/sec) and the MTU is 9000 (Jumbo Frames enabled)
As a test I tried using a "standard" E1000 vmnic (
e1000g0) and I was able to enable jumbo frames (9000 MTU) however reading the same file over NFS from VMware, I can only get a maximum of 0.124 GB/sec (0.992 gigabit/sec) which makes sense as the link speed on the E1000 is only 1000 megabit.
Why am I only seeing only about 1 gigabit/second transfer speeds between VMware ESXi 5.1 and my OpenIndiana Solaris VM?
UPDATE: After applying the performance tuning recommended by VFrontDe in his answer, I'm now getting an impressive 8-9 Gbit/sec between Solaris and a Linux VM with a
vmxnet3 adapter. I'm getting ~6 Gbps reading a cached file via NFS under Linux:
root@iperf:/mnt/public/ISO# dd if=Test.cdr of=/dev/null bs=1024k 5884+1 records in 5884+1 records out 6170824704 bytes (6.2 GB) copied, 8.44985 s, 730 MB/s
However Reading that same file on the VMware ESXi host, I get an abysmal 1.91 Gbps:
/vmfs/volumes/3d77318f-82c7901a/ISO # time dd if=Windows\ 7\ Home\ Premium.cdr of=/dev/null bs=1024k 5884+1 records in 5884+1 records out real 0m 27.85s user 0m 17.32s sys 0m 0.00s
What is causing Vmware ESXi's
vmknic to perform so poorly when
vmxnet3 is so much better?