Ubuntu 10.04 vms have sporadically high cpu usage on a hostA and not on HostB

Question

Now I know this is a loaded questions, so let me give you some background on the environment.

5 HP DL585 G7 esxi hosts in a cluster running esxi 5.1u1 running a bunch of ubuntu vms with various configs.

The issue is this, each host has anti-affinity rules set to separate 4 tomcat web servers on each of my hosts. Periodically on one host 2 out of the 4 web servers start showing major signs of cpu contention in the guest OS. IF you vmotion the servers to another host, it goes back down to normal CPU utilization. If you move it back to the host in question, its fine for a short period of time and spikes back up.

I have disabled the HP CIM pack that is loaded into this flavor of ESXi to see if this helps. Doesn't I have the C-States shut off on ALL the hosts. Still doesn't help The CPU Ready Time is good to ok The CSTP is good ( guest os's are balanced correctly )

I am at a loss, All the web servers configs are exactly the same and managed by puppets config. The load balancer is showing completely even loads to all web servers.

VMware support has had a ticket with me for 2 weeks with no direction or help on why this is happening.

I have not called HP support , but thinking about it.

Thoughts ? Ideas ?

Answer 1

So interesting conclusion after being on tech support calls for about a week and going through 3 different levels of support. So, what we failed to do in this specific cluster of ESXi 5.1 hosts was to shut off on our HP G7 385 servers the C-states! When you ESXTop into one of the hosts and choose "P" , it shows the columns for the power states. These columns are not supposed to be showing when the power scheme is set to high performance and the C-states set to OFF. What we thought we did was shut the Cstats off from the ILO connection setting powerstate to high performance. Do this in the bios. At certain points the processor utilization was pretty high, and the wait states on the host were , eh there but not to bad. The host was showing 40% of CPU utilization not being used. It felt like a restriction set on the host not to go over a certain % of CPU power.