Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I am running a production gentoo Linux machine, and recently there was a situation where the server hung in my co-located premises and when I got there I noticed that the server was hung on what appeared to be a kernel panic hang. I rebooted the machine with a hard reboot and was disappointed to find out that I could not find a shred of evidence anywhere on why the machine hung.

Is it true that when I do a hard reboot the messages itself will get lost or is there a setting I can do somewhere say in syslog-ng or maybe in sysctl to at least preserve the error log so that I can prevent such mishaps from happening in the future ? I am running a 2.6.x kernel by the way.

Thanks in advance.

share|improve this question

3 Answers 3

To log a kernel panic, using netconsole is way easier than using a serial link ;)

There is a useful entry about it on ubuntu wiki that can be used on other linux flavors... Helped me for instance with archlinux.

Note that netconsole can be also be setup at boot.

share|improve this answer

Generally in these situations I've set up a remote serial console and then made sure to log everything that appears on the console. You can do this by connecting the serial port of the machine to another machine via an rs-232 cable. This might be getting a little tricker these days as serial ports are gradually disappearing, but I think most servers still have them.

Then on that other machine, run conserver and configure console logging. This provides a good way to both log everything on the console and provide an alternate login path to the system.

If you don't want to set up and adminster another linux box, an alternative is to install a serial console server. Various companies such as Avocent, Cyclades, and Raritan make these boxes which provide a network interface to 2-48 serial ports. However this sort of thing will be more expensive than just setting up another linux box of course.

share|improve this answer
3  
In addition to Phil's answer, note that many (most?) *NIX systems will dump the kernel/RAM to the swap partition in the event of a panic, and copy the dump to /var/crash or /var/adm/crash upon reboot to allow you to attack it with a debugger and see where the panic took place. The serial console is still a great option though, as it lets you see what the last thing the server did before panic()ing was. –  voretaq7 Mar 10 '11 at 4:54

You can try replicating syslog messages to another machine. Maybe not all of them but only alerts and critical and disable caching for them.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.