Now I know this is a loaded questions, so let me give you some background on the environment.
5 HP DL585 G7 esxi hosts in a cluster running esxi 5.1u1 running a bunch of ubuntu vms with various configs.
The issue is this, each host has anti-affinity rules set to separate 4 tomcat web servers on each of my hosts. Periodically on one host 2 out of the 4 web servers start showing major signs of cpu contention in the guest OS. IF you vmotion the servers to another host, it goes back down to normal CPU utilization. If you move it back to the host in question, its fine for a short period of time and spikes back up.
I have disabled the HP CIM pack that is loaded into this flavor of ESXi to see if this helps. Doesn't I have the C-States shut off on ALL the hosts. Still doesn't help The CPU Ready Time is good to ok The CSTP is good ( guest os's are balanced correctly )
I am at a loss, All the web servers configs are exactly the same and managed by puppets config. The load balancer is showing completely even loads to all web servers.
VMware support has had a ticket with me for 2 weeks with no direction or help on why this is happening.
I have not called HP support , but thinking about it.
Thoughts ? Ideas ?