Monitoring dmesg output

Question

I find that when something goes wrong at a low level on one of my linux servers, I can see messages about it in the kernel ring buffer. These can be viewed from the command line using the dmesg command.

I am wondering if there is an easy way to get my servers to email me whenever something is added to the kernel ring buffer?

Right now I have a script that runs every hour, makes a copy of the output of dmesg and runs a diff against the file from the previous hour. Unfortunately this doesn't work so well, because as lines are appended to the end of the dmesg output, other lines are truncated from the beginning. Also, if I have alot of the same message, it just stops notifying me altogether.

• Is there a better way of doing this?

• Does anyone else think it is important to see these messages when they happen?

added

• is the information reported by the dmesg command the same as that in one of the logfiles? (If so, then the solution is easier than I thought)

Answer 1

There are a number of tools designed to gather this information and report it regularly.

I find the Lire tool (from the LogReport system) to be a good reporting tool, but you may also be interested in Logcheck and Logwatch. All are free software and can be installed directly from most major GNU/Linux package repositories.

Answer 2

This is a quick and dirty solution. You may get lots of mail. I'd advise adding some grep and/or grep -v commands. Of course you can use this technique for other log files, too. Add this command in your /etc/rc.d/rc.local or your system's equivalent (after having tested it from the command line).

sudo tail -F /var/log/messages | while read line ; do echo "$line"|mail -s Subject recipient; done &

edit: changed to capital F to make tail follow the file by name to be able to handle log rotation.

Answer 3

The following script will send any new entries to /var/log/kern.log to the root user.
Putting it in /etc/cron.hourly will send an email every hour, but only if there are new kernel messages.

#!/bin/bash

MAILTO=root
LOG=/var/log/kern.log

OFFSET_FILE=$0.offset
if [ ! -f $OFFSET_FILE ]; then echo 0 > $OFFSET_FILE; fi
OFFSET=`cat $OFFSET_FILE`
FILESIZE=`cat $LOG|wc -c`

# Check if log has been rotated
if [ "$OFFSET" -gt "$FILESIZE" ]; then
  OFFSET=0
  echo 0 > $OFFSET_FILE
fi
if [ "$FILESIZE" -gt "$OFFSET" ]; then
  tail -c+$OFFSET $LOG|sed "s/^/  /"|mail $MAILTO -s "new kernel alerts"
  echo $FILESIZE > $OFFSET_FILE
fi

Answer 4

There's a nice utility called cronolog(1) which reads a stream and splits it based on a date template. Example from one of my Apache config files:

CustomLog "|/usr/bin/cronolog -z0 /var/log/apache2/example.org/%Y/%Y-%m-%dZ.access_log" combined

This will split all Apache logs into a example.org/yyyy/yy-mm-ddZ.access_log format which makes life generally easier.

Now, how this can be used in your situation: Start a process which logs everything that happens in /var/log/messages :

tail -F /var/log/messages | cronolog -z0 /var/log/mylog/dmesg/%Y-%m-%d_%H.log &

This will split the output into files with the filename format

/var/log/mylog/dmesg/yyyy-mm-dd_hh.log

Every two minutes past the hour, run a script from cron that checks if there's generated a logfile the last hour. Something like this:

#!/bin/bash

# This script is executed every XX:02 from cron

LOGDIR=/var/log/mylog/dmesg
# Get date string five minutes ago
LOGDATE=$(date -u +"%Y-%m-%d_%H" -d "5 min ago")

if [ -e $LOGDIR/$LOGDATE.log ]; then
    mail -s "dmesg log for $LOGDATE" [email protected] <$LOGDIR/$LOGDATE.log
fi

If no file is found, no mail is sent. With this system, you'll also have backups of your log in the $LOGDIR directory.

Answer 5

If you do not have a kern.log and would like to create and keep one, edit /etc/syslog.conf and add the following

kern.* -/var/log/kern.log

Try 'man 5 syslog.conf' if you want more help on configuring syslogd.

Also swatch (http://sourceforge.net/projects/swatch/) is another tool that can easily be configured to watch log files.

Answer 6

StackKrish is on the right track but still doesn't send email. To do that you could use something like syslog-ng which allows logging to commands to a program with the the program() destination. If you don't want to do that you can get regular syslogd to log to a named pipe/fifo which then can be hooked up to a simple script that would send out the email.