Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I find that when something goes wrong at a low level on one of my linux servers, I can see messages about it in the kernel ring buffer. These can be viewed from the command line using the dmesg command.

I am wondering if there is an easy way to get my servers to email me whenever something is added to the kernel ring buffer?

Right now I have a script that runs every hour, makes a copy of the output of dmesg and runs a diff against the file from the previous hour. Unfortunately this doesn't work so well, because as lines are appended to the end of the dmesg output, other lines are truncated from the beginning. Also, if I have alot of the same message, it just stops notifying me altogether.

  • Is there a better way of doing this?

  • Does anyone else think it is important to see these messages when they happen?

added

  • is the information reported by the dmesg command the same as that in one of the logfiles? (If so, then the solution is easier than I thought)
share|improve this question
    
I think whether and which log files match dmesg is system dependent. You can configure kern.* messages to go to a file or files of your choice. –  Dennis Williamson Jun 12 '09 at 9:13
add comment

6 Answers

There are a number of tools designed to gather this information and report it regularly.

I find the Lire tool (from the LogReport system) to be a good reporting tool, but you may also be interested in Logcheck and Logwatch. All are free software and can be installed directly from most major GNU/Linux package repositories.

share|improve this answer
    
I think the asker wanted something that happened a little more often, but I don't really think there is anything. I use logcheck and have voted +1 –  David Pashley Jun 11 '09 at 23:21
add comment

This is a quick and dirty solution. You may get lots of mail. I'd advise adding some grep and/or grep -v commands. Of course you can use this technique for other log files, too. Add this command in your /etc/rc.d/rc.local or your system's equivalent (after having tested it from the command line).

sudo tail -F /var/log/messages | while read line ; do echo "$line"|mail -s Subject recipient; done &

edit: changed to capital F to make tail follow the file by name to be able to handle log rotation.

share|improve this answer
2  
Instead of tailing /var/log/messages (which may get log-entries from other syslog facilities) you could configure syslogd to log kern.* to another file and tail that. In addition, be mindful of logrotate which probably will break this when it does it's thing. –  Kjetil Joergensen Jun 12 '09 at 0:12
    
I edited my answer to handle the case of log rotation. Your suggestion regarding kern.* messages is a good one. I'd keep them going to /var/log/messages, too, though (just to be clear). –  Dennis Williamson Jun 12 '09 at 1:04
    
On my system (Debian), the output of "dmesg" is different from /var/log/messages. –  Brent Jun 12 '09 at 5:08
    
... It MAY be the same as /var/log/kern.log though - can anyone confirm this? –  Brent Jun 12 '09 at 5:14
    
@Brent - This Fedora 11 system doesn't have kern.log, but my Ubuntu desktop does. –  Dennis Williamson Jun 12 '09 at 8:05
add comment
up vote 1 down vote accepted

The following script will send any new entries to /var/log/kern.log to the root user.
Putting it in /etc/cron.hourly will send an email every hour, but only if there are new kernel messages.

#!/bin/bash

MAILTO=root
LOG=/var/log/kern.log

OFFSET_FILE=$0.offset
if [ ! -f $OFFSET_FILE ]; then echo 0 > $OFFSET_FILE; fi
OFFSET=`cat $OFFSET_FILE`
FILESIZE=`cat $LOG|wc -c`

# Check if log has been rotated
if [ "$OFFSET" -gt "$FILESIZE" ]; then
  OFFSET=0
  echo 0 > $OFFSET_FILE
fi
if [ "$FILESIZE" -gt "$OFFSET" ]; then
  tail -c+$OFFSET $LOG|sed "s/^/  /"|mail $MAILTO -s "new kernel alerts"
  echo $FILESIZE > $OFFSET_FILE
fi
share|improve this answer
1  
$OFFSET and $FILESIZE are in terms of characters (wc -c), but then the tail command is done in terms of lines (-n). They need to match. I would make it wc -l, but tail -c makes it where the outputs don't overlap by one line so that might be preferred. –  Dennis Williamson Jun 22 '09 at 9:56
    
Good catch Dennis, I will add -c to the script above. Thank you. –  Brent Jun 22 '09 at 13:21
add comment

There's a nice utility called cronolog(1) which reads a stream and splits it based on a date template. Example from one of my Apache config files:

CustomLog "|/usr/bin/cronolog -z0 /var/log/apache2/example.org/%Y/%Y-%m-%dZ.access_log" combined

This will split all Apache logs into a example.org/yyyy/yy-mm-ddZ.access_log format which makes life generally easier.

Now, how this can be used in your situation: Start a process which logs everything that happens in /var/log/messages :

tail -F /var/log/messages | cronolog -z0 /var/log/mylog/dmesg/%Y-%m-%d_%H.log &

This will split the output into files with the filename format

/var/log/mylog/dmesg/yyyy-mm-dd_hh.log

Every two minutes past the hour, run a script from cron that checks if there's generated a logfile the last hour. Something like this:

#!/bin/bash

# This script is executed every XX:02 from cron

LOGDIR=/var/log/mylog/dmesg
# Get date string five minutes ago
LOGDATE=$(date -u +"%Y-%m-%d_%H" -d "5 min ago")

if [ -e $LOGDIR/$LOGDATE.log ]; then
    mail -s "dmesg log for $LOGDATE" my@mail.com <$LOGDIR/$LOGDATE.log
fi

If no file is found, no mail is sent. With this system, you'll also have backups of your log in the $LOGDIR directory.

share|improve this answer
add comment

If you do not have a kern.log and would like to create and keep one, edit /etc/syslog.conf and add the following

kern.* -/var/log/kern.log

Try 'man 5 syslog.conf' if you want more help on configuring syslogd.

Also swatch (http://sourceforge.net/projects/swatch/) is another tool that can easily be configured to watch log files.

share|improve this answer
add comment

StackKrish is on the right track but still doesn't send email. To do that you could use something like syslog-ng which allows logging to commands to a program with the the program() destination. If you don't want to do that you can get regular syslogd to log to a named pipe/fifo which then can be hooked up to a simple script that would send out the email.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.