11

I am running a production gentoo Linux machine, and recently there was a situation where the server hung in my co-located premises and when I got there I noticed that the server was hung on what appeared to be a kernel panic hang. I rebooted the machine with a hard reboot and was disappointed to find out that I could not find a shred of evidence anywhere on why the machine hung.

Is it true that when I do a hard reboot the messages itself will get lost or is there a setting I can do somewhere say in syslog-ng or maybe in sysctl to at least preserve the error log so that I can prevent such mishaps from happening in the future ? I am running a 2.6.x kernel by the way.

Thanks in advance.

4 Answers 4

5

To log a kernel panic, using netconsole is way easier than using a serial link ;)

There is a useful entry about it on ubuntu wiki that can be used on other linux flavors... Helped me for instance with archlinux.

Note that netconsole can be also be setup at boot.

4

Generally in these situations I've set up a remote serial console and then made sure to log everything that appears on the console. You can do this by connecting the serial port of the machine to another machine via an rs-232 cable. This might be getting a little tricker these days as serial ports are gradually disappearing, but I think most servers still have them.

Then on that other machine, run conserver and configure console logging. This provides a good way to both log everything on the console and provide an alternate login path to the system.

If you don't want to set up and adminster another linux box, an alternative is to install a serial console server. Various companies such as Avocent, Cyclades, and Raritan make these boxes which provide a network interface to 2-48 serial ports. However this sort of thing will be more expensive than just setting up another linux box of course.

1
  • 4
    In addition to Phil's answer, note that many (most?) *NIX systems will dump the kernel/RAM to the swap partition in the event of a panic, and copy the dump to /var/crash or /var/adm/crash upon reboot to allow you to attack it with a debugger and see where the panic took place. The serial console is still a great option though, as it lets you see what the last thing the server did before panic()ing was.
    – voretaq7
    Mar 10, 2011 at 4:54
0

You can try replicating syslog messages to another machine. Maybe not all of them but only alerts and critical and disable caching for them.

0

I think there are three possible solutions for that:

  1. Configure kdump that creates a dump when the kernel panics (use "magic sysrq c" to force a panic). Probably the best solution if you really have a kernel panic.
  2. Configure a serial console (nowadays maybe netconsole). Good to see the famous "last words".
  3. Configure an external syslog server that receives the kernel messages. However take care that messages are send immediately (without delay). If the network stack crashes, the important messages might get lost, however.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .