Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm running an Ubuntu 10.04 (x64) as a web/mysql server.

The server became unresponsive to SSH, Ping, HTTP etc. and the technician with physical access to the machine sent me this screengrab here:

http://img442.imageshack.us/img442/389/img00062201012211332.jpg

from the connected monitor before he rebooted (and the situation is fixed). I'm not sure what log this information is kept in as I can't find the text after checking the logs after reboot.

Can anyone help me to investigate what happened to try and ensure it doesn't happen again?

Thanks

share|improve this question
    
Typically, kernel panics (like this) don't get logged in a file because at the point the error is logged on the screen the system is essentially no longer functioning. A common solution is to configure the system with a serial console, and then use a second system to record console output -- this gets you a record of kernel panics (and is occasionally useful if you shoot yourself in the foot with firewall rules or network misconfiguration). –  larsks Dec 21 '10 at 14:51

5 Answers 5

I have seen such crashes when servers ran with too high load/too many processes over a prolonged period of time. To generally check what is happening on your machine, I recommend to install a monitoring framework such as munin on your server - that will help the analysis in case this happens again.

share|improve this answer

Ok, that's a stack trace from the kernel. I'm not a kernel expert, but the cause involves interrupts, irq (interrupt) balancing, and probably PICs. This is more common on laptop hardware than server stuff though. The broken laptop solution is to boot with a kernel option of noapic.

share|improve this answer

The kernel panic may come from a bunch of reasons, usually either a module problem (a driver that doesn't fit your hardware) or a hardware problem.

In your case, if the problem is not repeatable it is more likely to be of hardware origin.
And it could be the memory (bad memory is not always easy to identify).

I would boot the server - and choose during the grub screen (just after booting) the "memtest86" option. The memory test needs to run several days continuously.
If after 3 days there is no error, the memory is maybe ok.

share|improve this answer
    
While memtest is a good suggestion, 3 days downtime testing something that might be faulty is (in my opinion) totally unacceptable. –  Ben Pilbrow Dec 21 '10 at 23:32
    
It took 2 days to reveal the memory was bad on one of our server (after painful investigation the memory was our last resort). The problem is, sometimes the memory behaves strange only with certain patterns, of after a number of passes. It takes time to show. –  ring0 Dec 21 '10 at 23:47

If you have no other information (as ring0 said, it won't be saved to disk somewhere), then there isn't anything else you can do.

If you want to be proactive, or it happens randomly a few more times, you can try LKCD to capture a core dump. http://lkcd.sourceforge.net/

I don't know how much RAM you have, but even trying memtest86 for a couple hours could be beneficial. It won't catch really rare errors obviously.

I also suggest you add kernel.panic=5 /etc/sysctl.conf. That will cause the server to reboot after 5 seconds automatically if the kernel hangs again.

Finally, I think you should always have some kind of lights-out management. Then you could log in and copy down the message and restart the server yourself.

share|improve this answer

This may sound a little off, but I had issues with Ubuntu x64 running on a server that was a 64 bit server. I had these same errors and subsequent "freeze" issues very often. It tried removing drivers, adding back drivers, spent hours looking through bugs, and nothing was helping. I finally got it resolved by installing a 32bit version of Ubuntu. It worked, I did not need 64 bit, so I let it ride. This is not a good solution if you need 64 bit, but it might give you a road to explore a bit. Maybe look up the server your running Ubuntu on and see if there are know compatibility issues around it. Good luck.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.