Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have absolutely no idea how did the server crashed, the only exceptional thing I found is the following graph from munin:

http://i.stack.imgur.com/Jm08n.png

Please don't tell me I need more RAM, as you can see, before the incident occurred, everything was stable. I just don't get why suddenly the server crashed nor why the memory demand was suddenly so high.

share|improve this question
    
The logs for that day would be helpful. Unfortunately logrotate only keeps 5 rotations' worth of backups, and rotates each day by default. –  Ignacio Vazquez-Abrams Aug 28 '10 at 6:31

2 Answers 2

up vote 0 down vote accepted

First check dmesg and system logs for any kernel panel or memory use. Looks like you have an application that is using all your memory. Try this script that will log your process list in a file and you will know what caused you the problem:

#!/bin/bash
mkdir /tmp/mem_log
while [ 1 ] ; do
   date "+%Y-%m-%d %H:%M:%S"
   ps aux
   sleep 60
done

and execute it like this:

nohup ./mem_log.sh > /tmp/mem_log/mem_log.log &

After the next server crash, check the log to see what process used all your memory. It is a memory problem, but not because you don't have enough memory, it's just a faulty process that causes this.

share|improve this answer
    
Will try that. Thanks. Just one question, if I save the log file in the /tmp/ directory, wouldn't it be deleted if I reboot the server (after it crashed)? –  TheOnly92 Aug 28 '10 at 9:59
    
yes, sorry, you are right, save it in your home dir. –  MihaiM Aug 28 '10 at 11:56

You may want to install psmon and make it report/kill misbehaving memory-hungry processes. Psmon logs / emails about events it reacts to, so that way you can easily find out what is the rebel process you have there.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.