How to log kernel panics without KVM

Question

My server is crashing and I can't find an answer why. It all started after my datacenter upgrade RAM from 16 GB to 32 GB.

I also found such logs in dmesg - they've started to show itself just before the first kernel panic:

EXT4-fs error (device md2): ext4_ext_find_extent: bad header/extent in inode #97911179: invalid magic - magic 5f69, entries 28769, max 26988(0), depth 24939(0)
EXT4-fs error (device md2): ext4_ext_remove_space: bad header/extent in inode #97911179: invalid magic - magic 5f69, entries 28769, max 26988(0), depth 24939(0)
EXT4-fs error (device md2): ext4_mb_generate_buddy: EXT4-fs: group 20974: 8589 blocks in bitmap, 54896 in gd
JBD: Spotted dirty metadata buffer (dev = md2, blocknr = 0). There's a risk of filesystem corruption in case of system crash.
EXT4-fs error (device md2): ext4_ext_split: inode #97911179: (comm pdflush) eh_entries 28769 != eh_max 26988!
EXT4-fs (md2): delayed block allocation failed for inode 97911179 at logical offset 1039 with max blocks 1 with error -5

This should not happen!!  Data will be lost
EXT4-fs error (device md2): ext4_mb_generate_buddy: EXT4-fs: group 21731: 5 blocks in bitmap, 60762 in gd
JBD: Spotted dirty metadata buffer (dev = md2, blocknr = 0). There's a risk of filesystem corruption in case of system crash.

My system is CentOS 5.8 64-bit with latest kernel 2.6.18-308.20.1.el5. How can I check what is the reason of kernel panic without having an access to the KVM ?

I have told my datacenter admins to check the memory in the server.

Answer 1

You can maybe see 'netconsole' package wich log by UDP on another machine the kernel logs in a brute mode (not syslog). On the server, you should install netconsole and ask it to export to a log server based on 'nc' by example. In case of kernel panic, all the informations are recorded on log machine and you can start to analyze what's happend