Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I have a VMWare ESXi 5.1.0 (build 1065491) host running a Solaris (OpenIndiana) VM as a guest. I have created a private vSwitch for NFS traffic between ESXi and Solaris. The Solaris VM has the VMware tools installed and has a VMXnet3 adaptor (vmxnet3s0) on the private vSwitch.

Reading from a file directly on the Solaris VM using dd I get speeds of up to 4.5 GB/sec (44.8 gigabit/sec) when reading a file (if it has been cached by my ARC/L2ARC)

Reading the same file over NFS from VMware, I can only get a maximum of 0.163 GB/sec (1.131 gigabit/sec)

Why is there such a difference? I expect some overhead due to TCP and NFS, as well as virtual networking, but that difference isn't what I'd call overhead, it looks like something's wrong.

The CPU on the Solaris VM is never more than 20% used. Load averages are low. Plenty of free RAM on the host (it's not swapping) and CPU utilization on the host is well below 50% (typically around 33%). On the guest VM, dladm show-linkprop vmxnet3s0 shows that the speed is 10000 (10 Gbit/sec) and the MTU is 9000 (Jumbo Frames enabled)

As a test I tried using a "standard" E1000 vmnic (e1000g0) and I was able to enable jumbo frames (9000 MTU) however reading the same file over NFS from VMware, I can only get a maximum of 0.124 GB/sec (0.992 gigabit/sec) which makes sense as the link speed on the E1000 is only 1000 megabit.

Why am I only seeing only about 1 gigabit/second transfer speeds between VMware ESXi 5.1 and my OpenIndiana Solaris VM?


UPDATE: After applying the performance tuning recommended by VFrontDe in his answer, I'm now getting an impressive 8-9 Gbit/sec between Solaris and a Linux VM with a vmxnet3 adapter. I'm getting ~6 Gbps reading a cached file via NFS under Linux:

root@iperf:/mnt/public/ISO# dd if=Test.cdr of=/dev/null bs=1024k
5884+1 records in
5884+1 records out
6170824704 bytes (6.2 GB) copied, 8.44985 s, 730 MB/s

However Reading that same file on the VMware ESXi host, I get an abysmal 1.91 Gbps:

/vmfs/volumes/3d77318f-82c7901a/ISO # time dd if=Windows\ 7\ Home\ Premium.cdr of=/dev/null bs=1024k
5884+1 records in
5884+1 records out
real    0m 27.85s
user    0m 17.32s
sys     0m 0.00s

6.2 GB in 27.85 seconds works out to be 1.91 gbps.

What is causing Vmware ESXi's vmknic to perform so poorly when vmxnet3 to vmxnet3 is so much better?

share|improve this question

1 Answer 1

It looks like there are some issues and bugs with the vmxnet3 adapter for Solaris. Please have a look at this very detailed post: http://blog.cyberexplorer.me/2013/03/improving-vm-to-vm-network-throughput.html The author was able to get much better throughput after some tweaking, but it's too much to repeat all steps here.

share|improve this answer
    
Thanks. This was almost it, but I need some tuning on the VMware side. I'm not getting ~8-9 gbps between VMs on the host, but still only ~1-3 gbps between the VM and the host itself... –  Josh Sep 14 '13 at 15:54

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.