Scene: 16-core VMware ESX 4.1 host. Presume that the host is not over loaded in any single resource at all. Storage latencies are very low, total memory utilization is below the physical ram amount, and not a single VM exhibits any %RDY time or significant pagefile activity.
A single-processor Windows 2008 R2 VM, fully up to date with recent VMware Tools, running PeopleSoft application tier (forgive my vagueness), WebLogic, Jrockit, etc. esxtop and vcenter show physical CPU use below 70%, physical memory below 80%, the guest machine is not exhibiting signs of significant disk pagefile activity.
Users report that the machine is not processing as many events as it "should", and when they try logging in, there is significant mouse and keyboard event lag through RDP. Everything about it just feels "slow."
A second CPU is added to the VM. %RDY time does not go up appreciably, so this is judged as a sane action. Perceived and measured application performance improves dramatically.
Everyone scratches their heads, shrugs, and are happy that its fixed. Admins who have read the VMware performance troubleshooting guide are perplexed.
What did we miss?