One of our servers recently experienced some file system corruption and our root file system was automatically remounted as read only. The steps I took to recover were:
- attempted to
remount > mount -n -o remount /this failed
- rebooted the server
- was prompted to perform a manual
fsck, there were 5 orphaned inodes which required fixing.
After performing these steps I was able to gain access and the file system was writable again. Unfortunately I don't have any informative logs as none were written or I would have included these.
One cause that was been suggested was that our database was too busy to write the data to disk properly and this caused the issue, the high level of cache memory was given as an indication that this might be the case. However I'm not sure about this as although cache is high we aren't using the swap at all (output of
$ free -m total used free shared buffers cached Mem: 2041 1879 162 0 62 1599 -/+ buffers/cache: 216 1825 Swap: 471 0 471
Is there any way I can diagnose the fault after it has happened? Does MySQL look like a likely candidate?
If not are there any steps I should take in the future if this happens again?