Today I had to fix 3 VMs with near identical file system errors all running Redhat 5 or 6. Basically the VMs had corrupted inodes and remounted / as a read-only file system, also corrupting several mysql tables and several mounted drives. Running fsck -A -y on each server solved the problems as well as manually repairing the corrupted MySQL tables.
I wasn't present when the corruption happened, but my understanding is that the corruption was simultaneous, whether or not is was a service crashing I don't know. I can't find anything in dmesg, messages, audit.log, mysqld.log to indicate the underlying cause. I'm not an expert server admin especially in Redhat (Ubuntu) but any sort of ideas or guidance to troubleshooting would be very welcome.
Underlying architecture is VMWare hypervisor.
I've tentatively suggested that a possible cause for all 3 servers to experience corruption could be because of an underlying fault with the data storage device.
Anyone had experience with something similar?