Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

My linux server (debian 5) stalled and had to be rebooted.

Please advise what to check in order to find out what happened. Usual suspects like /var/log/messages, syslog, and kern.log. Could you give me only a hint that load average skyrocketed to over 100 before failure? Don't know where to look further.

share|improve this question
6  
After server fault, check stack overflow. (Sorry, sorry.) –  MadHatter Feb 13 '13 at 9:55
    
Please give more information about that server hard/soft configuration and services. –  pistache Feb 13 '13 at 10:34
add comment

1 Answer 1

up vote 3 down vote accepted

You need to give more information about your server configuration, and post those logs. How do you know that the load average got high ?

The "load average skyrocketing to over 100 before failure" symptom can have many causes, but there are frequent ones.

First, what is the load average ?

I recommend looking at this post to get more information on the subject : http://superuser.com/questions/23498/what-does-load-average-mean-in-unix-linux

The load average is an indicator of how many processes are currently in the "waiting" state. A very high load average indicates ressource exhaustion, and a common cause of this ressource exhaustion is long (or infinite) waiting times for I/O operations to complete.

What could cause this ?

  • I/O Waiting can be because of a failing NFS mount, of a failing hard disk drive. You may have been the victim of an attack.
  • Attacks like Slowloris tend to exhaust the file handle pool, and can hang I/O operations on the server.
  • Pay attention to fork bombs too. If you suffered a fork bomb, and that your memory got exhausted, you may see traces about the "OOM Killer", a kernel worker that sacrifices processes to free up memory for the system when everything else have failed. On some setups, that OOM Killer can make the system reboot.

Where to look ?

This really depends on your system configuration and environment. You did not provide enough details in your question to answer to this particular question. Here are some ideas, though :

  • First, check all of your hard drives. Run full SMART tests on them.
  • If you have hardware RAID, check your RAID controller's log.
  • If you are hosting web services, check for bandwidth usage history, and requests count.
  • If you have remote network mounts (sshfs, NFS), check their reliability
  • If you are hosting user shells on your server, check their .bash_history (This can be faked.)
  • If that server is a virtual machine, the problem may have nothing to do with your server but with your virtual services provider.
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.