We have several ESXi servers in colocation and this one has been running along fine for some time until a few weeks ago when it threw a drive from its mirror and "halted" all of the disk access. The ESXi machine was still somewhat accessible from a shell and through the vSphere Client, I could see the hardware monitor that told me the array had lost a drive, but couldn't interact with the VMs. None would respond to a shutdown, etc. So I had the staff at the colo reboot the server when the ESXi server itself wouldn't reboot. Came up fine, had the drive replaced and the array rebuilt properly. All was well.
Then last night the same "freeze" happens, except this time there's no drive fault. Same story, can't access any VMs, can't shut them down, can't reboot the ESXi server remotely, have to have the colo staff power cycle the machine and it reboots fine. Before the reboot there was nothing in the ESXi event log at all, nothing in fact to indicate there even was a problem except for the fact that all of the VMs were inaccessible (ESXi believed them to be running and fine. Performance graphs were all flatlined though).
Both before and after the reboot, the hardware monitors report all hardware as perfectly fine (processors, memory, storage, fans, power supply, etc)
Has anyone seen behavior like this?