So we have a number of vm's running on an ESX 4.1 server for product testing. The ESX Server is at times under heavy load. We've been experiencing high CPU levels during some use cases, but we can't always duplicate this. If the ESX server as a whole is under heavy load could this cause guest machines to show high CPU usage?
To ask it a different way, if the guest machines require more cpu resources than the server has, how does this affect CPU usage as indicated by the OS and process?
One of the canonical sources for ESX performance tips is the Performance Troubleshooting Guide! (PDF)
In short: If your host is busy, and your VMs are reporting a high "ready-time," (which is a counter of when the VM is ready to perform work but no physical CPU can be found for the machine to be scheduled by the hypervisor) then the symptoms you are experiencing are not that in-guest CPU usage is high, but that the individual VMs are not getting enough physical CPU time to service their workload. This can manifest in-guest as appearing like high CPU use, but this is just an accounting mistake the guest is making because it is not aware of virtualization itself.
So check for ready time and if it is indeed high, find a way to either add more physical cpu to your machine/cluster or reduce use.
In-guest CPU monitoring has the potential to skew when the machine is idle, and also when it is busy. In one scenario, the idle process is not scheduled by the hypervisor. In the other, high use by many machines will result in some machines being starved of cpu, and hence guest-level monitoring should not be used without additional hypervisor-level monitoring.
In-guest CPU monitoring is a bad idea, it's almost impossible inherently for any guest to ever really know how much CPU time it's taking, on any hypervisor - the only real way to know is to look outside via the hypervisor. So yeah, a very busy hypervisor could easily make an in-guest CPU usage stat way off base yes - just don't trust those figures anyway.
