I seem to have a strange issue with my VMware cluster, where I get inconsistent SCP transfer rates. I have Cluster1 and Cluster2, physically in different regions. I need to transfer large vmdk from Cluster1 to Cluster2. Here is what I get:
- SCP from VMware host, directly to VMware host, no compression: 4MB/s
- SCP from VMware host, directly to VMware host, with compression: 0.5MB/s
- SCP from VMware host in Cluster1, to one of the virtual machines running in Cluster1: 30MB/s
- SCP form virtual machine in Cluster1 to VMware host in Cluster2, with compression: 15MB/s
- SCP form virtual machine in Cluster1 to VMware host in Cluster2, no compression: 5MB/s
- SCP form virtual machine in Cluster1 to virtual machine in Cluster2, with compression: 20MB/s
- SCP form virtual machine in Cluster1 to virtual machine in Cluster2, no compression: 7MB/s
Testing network bandwidth with iperf shows that I have 200-300Mbps between the locations consistently. Network connection in both clusters, as well as internet is gigabit.
What would cause VMware to limit CPU usage for compression, and what would limit it's transfer rates outside of the cluster?
PS: inside the cluster I am going through public IP, and guest VM is on a different host. So theoretically VMware shouldn't know that the transfer is somewhat local.
EDIT: Cluster1 is 4.1, Cluster2 is 5.0; Tried FastSCP, and getting the same result as I see with direct SCP with compression: about 0.5MB/s.
EDIT 2: Increased system resource allocation on VMware hosts to the levels that VMs are getting and beyond. Only change is SCP with compressing from host to host is not up from 0.5M to 4M, just like non-compressed transfer.
Still puzzling why would host-to-host transfer be slower though.
EDIT 3: after adding more resources, was able to achieve 4~10MB/s transfer speeds across data centers. Even though it is lower than 20~40 that the network should be capable of, I'll just have to live with that. Although if anyone has any other ideas - I'm eager to try them :)