Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have absolutely no idea how did the server crashed, the only exceptional thing I found is the following graph from munin:

enter image description here

Please don't tell me I need more RAM, as you can see, before the incident occurred, everything was stable. I just don't get why suddenly the server crashed nor why the memory demand was suddenly so high.

share|improve this question
The logs for that day would be helpful. Unfortunately logrotate only keeps 5 rotations' worth of backups, and rotates each day by default. – Ignacio Vazquez-Abrams Aug 28 '10 at 6:31
Check the logs, what does log say? check your hard drives for any bad sectors (this could lead to crash)! I see that committed memory (allotted to applications) is through the roof. Committed memory is, essentially, all the memory which has been allocated by applications. You may want to increase the swap space. see this:… – pbu Mar 7 at 15:33
up vote 0 down vote accepted

First check dmesg and system logs for any kernel panel or memory use. Looks like you have an application that is using all your memory. Try this script that will log your process list in a file and you will know what caused you the problem:

mkdir /tmp/mem_log
while [ 1 ] ; do
   date "+%Y-%m-%d %H:%M:%S"
   ps aux
   sleep 60

and execute it like this:

nohup ./ > /tmp/mem_log/mem_log.log &

After the next server crash, check the log to see what process used all your memory. It is a memory problem, but not because you don't have enough memory, it's just a faulty process that causes this.

share|improve this answer
Will try that. Thanks. Just one question, if I save the log file in the /tmp/ directory, wouldn't it be deleted if I reboot the server (after it crashed)? – TheOnly92 Aug 28 '10 at 9:59
yes, sorry, you are right, save it in your home dir. – MihaiM Aug 28 '10 at 11:56

You may want to install psmon and make it report/kill misbehaving memory-hungry processes. Psmon logs / emails about events it reacts to, so that way you can easily find out what is the rebel process you have there.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.