New answers tagged fsck
To explicitly answer your questions: Q: What hope do I have of repairing these systems? A: Quite good since the fs is readable, but I'd plan on abandoning the current hard drive(s). Q: Separately, what went wrong? A: Unless you can definitively diagnose a hardware error (likely), you'll probably never know. You don't mention if you've looked for low-level ...
You shouldn't even try to do fsck on a specific partition, because you're using software RAID and LV on top of it. So the filesystem is on other/"higher" level. But you could use badblocks on a specific partition if it's not used by OS (/dev/sdb3 in this case). Then you could (and you should) re-add your partition to the RAID, if the device state is OK, by ...
Run a memcheck on your RAM. If you have bad RAM, it will cause random corruption in your filesystem. Run a health check on your hard drives and mainboard
once in a while after a reboot I found fsck would fail on OS disk and booting becoming impossible on many of my servers. Are you doing graceful reboots/shutdowns on these systems? If you are doing so (meaning filesystems get cleanly un-mounted), and you are still seeing corruption, then it's likely that the underlying storage has issues. What ...
I will allow others to answer who may have better ideas then me on this but after running e2fsck several times after umount, mount, start, repeat... I managed to bring all the containers back up finally. In fact, I think you can skip the umount process as one started right away and I forgot to umount it.
Top 50 recent answers are included