Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

One of my servers, which runs in a german data center "hangs" every night, but i cant find out why. No errors are found in the /var/log/messages and /var/log/syslog.

The server responds to ping, but all services are down (ssh, apache, ...). After a reset everything runs normal.

A hardware test has been performed. It looks like being a software issue.

share|improve this question
    
Is it possible to log into local console when the server "hangs"? Examine top output. May be some process just takes almost all CPU time and all network services just get connection timeout. –  HUB Jun 20 '11 at 16:27
    
Its possible to log into a local console, but a little bit complicated. This had been my next step. I now will log top output every minute like Eduardo suggested. –  Martin Schlagnitweit Jun 20 '11 at 18:10
    
It seems to be a kernel panic. At the local console i can see this. But into which logfiles should this be written? –  Martin Schlagnitweit Jun 21 '11 at 5:41
    
Is there "/var/log/kern.log" file? –  HUB Jun 21 '11 at 9:22
    
In the kern.log file there are some extries like this: Jun 25 14:05:39 solunic kernel: [369632.475072] php-cgi[15194] general protection ip:6914c9 sp:7fffaf0f84d0 error:0 in php5-cgi[400000+6f9000] Today the server crahsed again at 16:00 CET, but the last error in kern.log was one hour before this. –  Martin Schlagnitweit Jun 25 '11 at 15:27

3 Answers 3

I'd leave some light profiling commands logging to files, so you can get an inside look on what went wrong after the fact. For example:

nohup top -b -d 60 >> top.log & # runs every 60 seconds
nohup vmstat 5 >> vmstat.log &
nohup iostat 5 >> iostat.log &

nohup is there so they aren't killed when you lose connection to the server. You can also use screen for that.

A more robust alternative to the last two commands would be to setup sar.

share|improve this answer
    
I bet you meant to use ">>" instead of ">", right? –  mfinni Jun 20 '11 at 17:08
    
It'd be better on some cases, but it's not necessary - each command keeps running and so opens/overwrites the output files only once at start. But I'll change it just in case, thanks! –  Eduardo Ivanec Jun 20 '11 at 17:16
    
If those keep running after the "symptom" happens, then the output files will get over-written. That's why I'm suggesting the change. –  mfinni Jun 20 '11 at 17:40
    
Oh wait - you mean the commands continue to run in the background, with the output redirected only once? Neat trick. –  mfinni Jun 20 '11 at 17:46
    
Well, actually I meant the commands to be executed separately, and yes - they do keep running/cycling by themselves. But I'll add nohup and & because I wasn't that clear, thanks. –  Eduardo Ivanec Jun 20 '11 at 18:58

When I have seen issues like this, it usually ends up being a problem with a cron job.

Check your syslog for cron jobs running at the same time of day that the server hangs. Also, check your root crontab (crontab -e) and jobs in /etc/cron.daily for anything that might be responsible.

share|improve this answer

Sounds like the random crashing could be caused by to faulty hardware. Have the hosting company see if there are any errors on POST or on the servers LCD. If it is a dell server you might want to install open manage which will tell you if any hardware is faulty. In my experience, a faulty memory dimm can cause random server reboots. Depending on what type of hardware you are running, it should be possible for your hosting to do a chassis swap on the server if the issue continues.

share|improve this answer
    
-1 for not reading the question –  Michael Lowman Jun 20 '11 at 18:39
    
An extensive hardware test has been performed by the hosting company. –  Martin Schlagnitweit Jun 21 '11 at 5:42

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.