Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Twice now in the last 4 days at some point during the night websites go down because the server is unable to connect to the database. At this point everything else is still running (apache ect) just the database is dead.

When I log into ssh with root to investigate, I have read only permissions everywhere which is what I suspect is the cause of the mysql server dying.

I've checked the mysql logs, system logs, basically every log file I can find, nothing indicates an error anywhere around when the problems starts (or even during the entire day). It is like a switch is just flipped, I then restart the system and things are fine again... until a few days later?

There was 2G of free ram the last time this happened, 1.5G free the first time. Minimal cpu usage (< 30%).

Any ideas?

share|improve this question

2 Answers 2

up vote 3 down vote accepted

Disk errors are one possible cause of the "I have read only permissions everywhere" condition. Some types of hardware or kernel-level disk faults can lead to an inconsistent and corrupted filesystem, so the kernel will protectively force the filesystem into "read-only" mode when it detects such a fault. If the disk containing your root filesystem has a hiccup, anything trying to write to the disk would start to see permission errors. Programs that don't need to write to the disk (like Apache or SSHd) will probably continue to work just fine.

The next time the database fault happens, check your kernel log message buffer BEFORE YOU REBOOT for any indications of a disk error. You'll probably have to use the 'dmesg' command, because if your '/var/log' directory is part of the root filesystem, the syslog daemon won't be able to write the error messages to the '/var/log/messages' file on the disk. Also, the contents of the kernel log buffer will be lost when you reboot, so you might want to use 'ssh' or 'scp' to copy that data elsewhere.

share|improve this answer
    
+1. I've had similar problems. It's always occurred when the kernel detected a problem with the file system (xfs in my case). The kernel put the file system into a read-only state. Only a reboot will restore the file system to read-write. –  David Mar 3 '10 at 17:39
    
I have xfs running - maybe it is the same thing for me? I'll be certain to try the dmesg command next time it happens! –  savageguy Mar 3 '10 at 18:09
    
David, did you ever correct the problem? –  savageguy Mar 3 '10 at 18:31
    
I have not had the problem since stopping the xfs service so I've marked this as the solution. Thanks! –  savageguy Mar 10 '10 at 15:43

I had the same issues. It looked like the filesystem was being set to read only for no reason, there (obviously) wasn't anything in the log files.

It took months to get this resolved with our "we don't support linux" hardware vendors.

I eventually found a firmware bug on the RAID controller. It would degrade the array when it hit the bug and the ext3 filesystem on top of the RAID was set to read only by the kernel. A firmware upgrade fixed the bug and the server has gone from flaking out every few days to being rock solid.

On a side note: If you are running a MegaRAID card, install the MegaCli package and use it to check the event logs.

I found this document very useful.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.