ESX hosts crash within same vcenter cluster

Question

I have a vcenter cluster of 12 ESX hosts (ClusterA) and another cluster of 3 ESX hosts (ClusterB). All of these are a mix of poweredge r620s and r630s.

Some of the hosts have hardware errors that can be seen in the iDRAC logs and front LCD screen such as:

• CPU machine check error
• Correctable memory error rate exceeded As expected, this is causing those hosts to be unavailable (Not responding) in the cluster.

Fixing these hardware errors usually involves these steps:

1. power off
2. remove network cards
3. power on and wait for successful boot to OS
4. power off
5. place the same network cards back in
6. power on It's strange to me that this would fix CPU & memory errors, but that's what happens consistently.

ClusterB is fine - no problems ever. The real problem I'm facing is that when I fix a couple hosts from ClusterA, 1-3 other random hosts in ClusterA will crash within a day or two. After those initial 1-3 crashes, if I leave things alone, no more hosts crash afterwards for weeks. This puts me back to where I started and I've observed this behavior several times now.

Any ideas on what to check?

Answer 1

The R620 / R630 are very old so for starters they may just be on their last legs and have legitimate hardware failures. That said, things that can potentially cause those sorts of problems:

• Power - try statically setting P states and power to maximum output in the iDRAC. You can sometimes see odd processor behavior due to power consumption saving measures if things aren't working properly
• Temperature - I have also seen those errors in datacenters running outside normal operating temperatures
• BIOS patches - those are some old servers but, for example, if there were an issue with p-states or something else of that nature that's the best way to receive a fix, but I assume Dell support would have already told you to about that.
• Other hardware anomalies? Anything strange in the iDRAC logs?