Scene: 16-core VMware ESX 4.1 host. Presume that the host is not over loaded in any single resource at all. Storage latencies are very low, total memory utilization is below the physical ram amount, and not a single VM exhibits any %RDY time or significant pagefile activity.

A single-processor Windows 2008 R2 VM, fully up to date with recent VMware Tools, running PeopleSoft application tier (forgive my vagueness), WebLogic, Jrockit, etc. esxtop and vcenter show physical CPU use below 70%, physical memory below 80%, the guest machine is not exhibiting signs of significant disk pagefile activity.

Users report that the machine is not processing as many events as it "should", and when they try logging in, there is significant mouse and keyboard event lag through RDP. Everything about it just feels "slow."

A second CPU is added to the VM. %RDY time does not go up appreciably, so this is judged as a sane action. Perceived and measured application performance improves dramatically.

Everyone scratches their heads, shrugs, and are happy that its fixed. Admins who have read the VMware performance troubleshooting guide are perplexed.

What did we miss?


CPU Ready shows how busy the host CPUs are rather than the guest. High %RDY would probably correspond to high CPU utilization on the VM but not necessarily and if you have a lot of VMs it might not be the one in question with the issue.

From the VMWARE communities:

For every cycle of time the server has a determined number of available CPUS (including cores) to provide out to VMS. Lets say you have 2 quad core cpus. THen you have 8 available CPUS. So if you have 8 VMs running and each of them only has one vcpu then all of them get to run during that cycle. Problem occurs when you have VMs demanding more CPUs than you have available. For example if you have 6 VMS with 1 vcpus. each and 1 VM with 4 vcpus. Obviously they can't run all at the same time. VMware has to try to accomodate all the vCPUS of the 1 VM at the same time.. which means that for that one cycle either 2 VMS are on standy or the multi vCPU VM is on standby.

Certain applications that are multi-threaded will use multiple vCPUs, which will help with performance.

In your case, were you actually doing perfmon stats/logging over time to get a baseline and then doing one after the vCPU was added? That would probably give you a better indication then using the built in VMWare tools of where the differences really lie, especially at the application/process level.

  • None of the metrics we were familiar with showed CPU-bound resources. Are there cases of applications where even if physical CPU does not reach 100% utilization, a second CPU will help? – Yolo Perdiem Dec 20 '12 at 17:17
  • I don't know for certain on your apps. That's where doing analysis on the app itself is important. Since adding the 2nd vCPU helped, it could have been contention for the physical proc, it could have been that the app allows for multi-threaded processing, etc. Using perfmon would have helped isolate this, along with process explorer. – TheCleaner Dec 20 '12 at 17:20
  • @evilensky Did you do some benchmarking or resource monitoring WITHIN the VM? – John Siu Dec 20 '12 at 22:07
  • @JohnSiu The users reported high CPU use within the guest, sure, but esxtop only showed 60-80% physical utilization. – Yolo Perdiem Jan 8 '13 at 14:51
  • @evilensky High CPU utilization maybe due to I/O delay. Check IO performance, especially on IO wait time. I believe you have to use performance monitor or 3rd party tools to investigate that, task manager don't have that information. You can also try to monitor that on the host level and see if host IO, especially storage, is max. out. – John Siu Jan 8 '13 at 15:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.