Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a linux machine dedicated to MySQL server with a pretty high load. Today I woke up and was terrified to see that database server is down. I could connect to it via SSH, but it was responding with bus error to each and every command:

[root@r1304 home]# ls
Bus error
[root@r1304 home]# tail /var/log/messages
Bus error
[root@r1304 home]# reboot
Bus error
[root@r1304 home]# free -m
Bus error
[root@r1304 home]# chkdisk
Bus error

I went to Data Center and did a hard reset, which seemed to help, but after a half an hour situation reapeated and now I can't even connet via SSH anymore.

Any ideas what this could be? how to diagnose such a problem and what are possible fixes?

Server has 32 GB RAM, 2xSSD drives with software RAID

UPDATE According to Zabbix, when MySQL died, number of processes stated to increase drammaticaly, until I did a hard reset. What could those be?

screenshot Number of processes

share|improve this question
    
Those are processes piling up waiting for I/O. Something is happening at the storage layer. –  ewwhite Dec 8 '12 at 23:47
    
What was the issue? –  ewwhite Dec 10 '12 at 10:32
    
@ewwhite, well, the hard drive completely died. Hello, kingston SSD. I'm bying a new one and restoring from a backup. –  Temnovit Dec 10 '12 at 19:41

1 Answer 1

up vote 6 down vote accepted

Your storage system seems to have failed. You can't read from disk at this point.

Can you see the messages in the kernel ring buffer? Look at the output of dmesg if the command even works at this point.

Do you have any backups or replication enabled?

share|improve this answer
    
Thank you for your answer. Yes, I have cold backups and replication on another server. Does this mean I have replace the hard drives? Unfortunatly, nothing works for me at this point. –  Temnovit Dec 8 '12 at 13:50
    
If you have data elsewhere, I would reboot and try to examine the logs in /var/log/messages as soon at the system comes up. Look for indicators of filesystem or hardware failure. –  ewwhite Dec 8 '12 at 13:52

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.