Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a server running on Centos 5.2 and is there any a better way to know why the server crashed or what it's doing at that time?

I am sorry I am a newbie and any help is appreciated~ Thanks

share|improve this question

6 Answers 6

up vote 8 down vote accepted

If you have experienced a kernel panic, you can set up a remote kernel console to capture all the data that might be lost on the local console (especially if the crash is from a non-maskable interrupt, which tends to reboot the system).

On the system that you expect might crash:

/sbin/modprobe netconsole netconsole=6666@10.1.1.16/eth0,6666@10.1.1.17/00:19:BB:31:B8:0E
  • 6666 is an arbitrary port number
  • 10.1.1.16 is the IP address of the local interface to send via
  • eth0 is the name of the local interface to send via
  • 10.1.1.17 is the IP address of the remote interface to send to
  • 00:19:BB:31:B8:0E is the MAC address of the remote interface to send to

On the remote system, run (this requires that you have netcat installed):

nc -l -p 6666 -u | tee capture.file

This will capture all kernel output on the remote system. This runs at a much lower level (the same point in the kernel that writes to /dev/klog), so you may see the very last bit of information that the kernel outputs when it panics even if syslog et. al have stopped operating.

share|improve this answer
    
I know there is a way of dumping kernel core when crashing. But is it the only most often used method to do so? any other way? I mean some other feasible and effective solutions –  Mickey Shine Jul 24 '09 at 9:47
    
+1 on the netconsole. I've been meaning to write a blog post about this for some time since IMO it's one of the best kernel debugging tools you can have. It's helped me solve numerous problems. –  Kamil Kisiel Jul 24 '09 at 21:41
    
+1 on the netconsole, too. I've been searching for such a solution some time ago and didn't find any appropriate solution. –  Manuel Faux Jul 26 '09 at 14:47

try starting process accounting

/etc/init.d/psacct start or /sbin/chkconfig psacct on (for autostart on boot)

then use lastcomm(1) to see what was running when.

or try installing atop, it will log your machine memory and process state every 10 minutes so you can get an idea what was going on.

atop -r /var/log/atop/atop_YYYYMMDD and then use t and T keys to go forwards and backwards

in 99% of the cases it is clear from those two exactly what was going on

share|improve this answer
    
+1 Very interesting. I must admit I had never used lastcomm et al. before. –  wzzrd Jul 24 '09 at 14:55
    
Keeping track of what has been running in useland may give you a clue, but it's not really going to help. Getting a core and looking into it is the only way to be sure. When the panic happens, the kernel isn't going to spend anytime writing out log files, the important log, the kernel ring buffer, isn't going to get dumped to the filesystem. –  goo Jul 24 '09 at 18:45
    
Thank you. I will try this way and I guess it might be some easier than kernel dumping~ –  Mickey Shine Jul 25 '09 at 5:37
    
Geoff, the goal here is to first try easier ways and bring in the heavy artillery only if necessary. In the last 15 years I had servers crashing front and center and only one time I had to resort to kernel dumps or network logging. –  Aleksandar Ivanisevic Jul 27 '09 at 12:38

Have you checked /var/log/dmesg, /var/log/messages, and /var/log/syslog?

share|improve this answer
    
yes, but there is nothing helps. ( I have no /var/log/syslog on my server, is that normal?) –  Mickey Shine Jul 24 '09 at 9:14
3  
Yes, it is. On Centos and RHEL you have /var/log/messages and /var/log/dmesg, no /var/log/syslog –  wzzrd Jul 24 '09 at 9:26
    
Normal logs are only likely to be of use if there is a piece of hardware that is failing slowly. As there is nothing there, this is a sudden failure, so it's either a kernel problem, tainted kernel mod problem or sudden hardware failure. –  goo Jul 24 '09 at 18:49

What sort of crash? Everyone's recommendation about dmesg / messages logs are good. If it is just 'shutting off' before it has the chance to log anything, I would guess it might be overheating or there is a power supply problem.

If this is the case, it might be helpful to go to the hardware logs if they exist. If you use Dell servers, Dell support can give you Linux tools to access these logs. Other vendors might provide similar functionality.

You might also check the memory with memtest86.

share|improve this answer
    
+1 for memtest recommendation. I've been bit by bad ram so many times... –  Josh Jul 24 '09 at 22:01

Collecting a core over the network is probably overkill, you can dump it locally. This is a guide for setting up and testing kdump. If you follow the instructions and still can't get a dump created locally then you should then move on to capturing over the network..

Of course once you have a core dump, you'll need to do some analysis on it using the crash utility. You'll need to install the right kernel-debuginfo rpm for your running kernel and then invoke crash - you should get the general gist from the whitepaper. If you can get it open the first thing you should look at is the log - scroll down to the bottom and you should get some clues as to what is going on at the time the crash occurs.

share|improve this answer
    
I like this answer, but core dumping over the network is not much trouble to set up than dumping locally. Don't know why I suggested the network dump though; probably needed some coffee at the time :P Anyway, +1 for pointing at the guide and whitepaper. –  wzzrd Jul 24 '09 at 14:37

You could configure the machine to do a kernel core dump over the network, but you'd still need someone skilled to look into that.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.