0

My linux server (debian 5) stalled and had to be rebooted.

Please advise what to check in order to find out what happened. Usual suspects like /var/log/messages, syslog, and kern.log. Could you give me only a hint that load average skyrocketed to over 100 before failure? Don't know where to look further.

2
  • 6
    After server fault, check stack overflow. (Sorry, sorry.)
    – MadHatter
    Feb 13, 2013 at 9:55
  • Please give more information about that server hard/soft configuration and services.
    – pistache
    Feb 13, 2013 at 10:34

1 Answer 1

4

You need to give more information about your server configuration, and post those logs. How do you know that the load average got high ?

The "load average skyrocketing to over 100 before failure" symptom can have many causes, but there are frequent ones.

First, what is the load average ?

I recommend looking at this post to get more information on the subject : https://superuser.com/questions/23498/what-does-load-average-mean-in-unix-linux

The load average is an indicator of how many processes are currently in the "waiting" state. A very high load average indicates ressource exhaustion, and a common cause of this ressource exhaustion is long (or infinite) waiting times for I/O operations to complete.

What could cause this ?

  • I/O Waiting can be because of a failing NFS mount, of a failing hard disk drive. You may have been the victim of an attack.
  • Attacks like Slowloris tend to exhaust the file handle pool, and can hang I/O operations on the server.
  • Pay attention to fork bombs too. If you suffered a fork bomb, and that your memory got exhausted, you may see traces about the "OOM Killer", a kernel worker that sacrifices processes to free up memory for the system when everything else have failed. On some setups, that OOM Killer can make the system reboot.

Where to look ?

This really depends on your system configuration and environment. You did not provide enough details in your question to answer to this particular question. Here are some ideas, though :

  • First, check all of your hard drives. Run full SMART tests on them.
  • If you have hardware RAID, check your RAID controller's log.
  • If you are hosting web services, check for bandwidth usage history, and requests count.
  • If you have remote network mounts (sshfs, NFS), check their reliability
  • If you are hosting user shells on your server, check their .bash_history (This can be faked.)
  • If that server is a virtual machine, the problem may have nothing to do with your server but with your virtual services provider.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .