Our application runs 50% slower on a Xeon X5650 under ESX than on a bare-metal E5450.

A test task on the physical servers takes 17 minutes.

The same task on the virtual servers takes 25 minutes: 50% longer.

From everything I can tell, this should be impossible; the 5600 series is supposedly 10-20% faster than the 5400 series for single-threaded processes at the same clock speed, and virtualization overhead should similarly low for single-thread CPU-bound workloads. Performance should at least break even, shouldn't it? But instead of having equal or better performance, performance is cut by 1/3.

UPDATE: Solved. The same task on the (fixed) virtual servers takes 14 minutes: 15% faster.

It was the RAM configuration. The 50% performance drop was because the ESX host system memory installed wrong and only providing around half of the full possible bandwidth. For the CPU & memory-bound process, that loss of bandwidth translated into 50% worse performance than expected.

Application performance is now smack in the middle of the 10-20% improvement we originally expected.

There are two physical Windows Server 2003 R2 systems which run an application which consists of a single-threaded calculation run on one server (32-bit), talking back and forth to an SQL Server 2005 database on the other (64-bit).

Both physical boxes are single-CPU E5450 with 4GB RAM @800Mhz. The calculation server never uses more than 1.5GB physical memory, and the SQL Server never uses more than 2.5GB. CPU utilization on the calculation server never goes over ~15% (around 50% of a single core). CPU utilization on the DB server never goes above ~25% (a fully-utilized single core).

The physical ESX 4.1 hosts are dual-CPU X5650 with 64GB RAM @1333Mhz. The virtual machines are given 4 cores and 4GB RAM each, to mirror the physical environment. The test was made with a single VM running on each physical host, as well as with both running on the same host.

Interestingly, we get very nearly the same 25-minute test results on another pair of ESX servers using X5550 CPUs and RAM @1066Mhz.

Also, test results in the virtual systems do not vary by more than 10% either way giving the VMs 1, 2, or 4 CPUs, or 1, 2, 4, or 8GB RAM. There is very little network or disk activity, and as far as I can tell the process should be CPU bound.

Tests have been run using both local 15K SAS disks on the separate hosts, as well as to an gigabit iSCSI SAN also with 15K disks. There is negligible different in results for different storage.

From everything I can tell, the Xeon 5600-series should be 20-50% faster than the 5400-series for single-threaded workloads. Even considering that the X5650 is a 2.67GHz part and the E5450 a 3GHhz part, if per-core performance was equal at the same clock speed you would still expect to see at least 90% of the performance instead of 67%. This doesn't even take into account the fact that the memory clock is nearly twice the speed.

It should be said that I've done several virtualization projects in the past, and never seen a anything close to a 50% performance degredation even using the SAME physical CPU cores, let alone cores two generations newer with faster memory.

Any ideas on possible causes or any configuration settings I should check?

  • The virtual setup use a virtual switch now though? – Steve-o Aug 30 '11 at 9:55
  • 1
    I can't give you any advice except that some applications just doesnt work in virtualized enviroments. We keep a physical box for MS Search server for a very good reason.. – pauska Aug 30 '11 at 10:23
  • @Steve-o Not a distributed switch, if that's what you mean; standard vSwitch using bonded gigabit ports on both sides. But total network traffic is only about 200KB during the entire 17- or 25-minute test. – ryandenki Aug 30 '11 at 10:32
  • You might be subjecting yourself to high latency, the only ways to test are VMs on separate machines or temporarily adding a physical NIC for each host. – Steve-o Aug 30 '11 at 10:56
  • Sorry if it wasn't clear; these are being tested on separate machines connected by physical NICs. Have edited to make that clear. – ryandenki Aug 31 '11 at 0:14

Depending on the type of virtualization, 5% overhead is pretty much best case scenario. With full paravirtualization, you can achieve such overhead on IO-light workloads quite easily. With hardware-assisted virtualization (technology used by VMWare), it is possible to achieve such a low overhead on IO-light workloads on an hypervisor with few VM. With full virtualization (no CPU extensions), 5% overhead is pretty much a dream.

Keep in mind this can depend on very many factors. Virtualization tends to add a significant amount of latency between disks and the guest OS. This will increase IO wait and therefore load averages while keeping CPU usage rather low. If your storage is on the lower side of the IOPS scale, this will have a very big impact. If you are using network storage, this will almost always add latency due to having to access a network for each IO instead of just accessing an internal bus.

Virtualization can also add extra network latency if you use special network configuration modules such as virtual switches but this usually is not very significant.

Virtualization tends to add many extra interrupts which are required to switch from a VM to another. Depending on the scheduler of the hypervisor, this can be significant. There isn't much you can do about this since it is just due to the nature of virtualization. But it is something to keep in mind as a justification to lower performance.

Due to the single-threaded nature of your application, having more cores will yield no significant performance improvement. Both CPUs have similar frequencies but you will notice the X5650 has a slower frequency without "Turbo Boost". You may want to check that feature is compatible/enabled with your setup.

33% overhead on IO intensive workload is, I find, not so bad. Try separating the storage for your two VMs and see if it helps.

  • OK, maybe you don't expect to see only 5% overhead, but do you expect to see 50% degradation on 5-years newer hardware, running 1 VM per ESX host? – ryandenki Aug 30 '11 at 10:24
  • CPUs are pretty much similar looking rapidly at the specs if you forget the difference in the number of cores. You have not even mentioned the type of storage you are using... Virtualization does not like IO-intensive workloads, this is nothing new. – Antoine Benkemoun Aug 30 '11 at 10:31
  • I've run the tests on 15K SAS local storage, separate per host, and also over a shared iSCSI SAN (also 15K). No significant different between runs on either type of storage. According to the Windows performance counters, total read bytes during the test is about 5MB, total written about 150KB. VMware statistics look about the same. – ryandenki Aug 30 '11 at 10:37
  • I don't expect the extra cores to have any difference except maybe in not interrupting the processing thread with any other background tasks that may pop up, and testing shows no such impact. Your comment about the Turbo Boost mode is interesting though--there is not a blip on power consumption at all, which I would expect if that were kicking in. I'll have to reboot and verify the BIOS settings for that. Thanks for the idea. – ryandenki Aug 30 '11 at 10:41
  • Are you using vmware-tools ? – Antoine Benkemoun Aug 30 '11 at 10:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.