12

I'd like to be able to get an email notification whenever syslogd logs something of, say err priority or higher. Assume this is a BSD compatible syslog daemon.

  • Can this be accomplished?
  • Should I use a named pipe to a shell script?
  • What other possible solutions are there?

4 Answers 4

6

Since the link no longer works, I've changed it to the Internet Archive and quoted a portion of the article here:

named-pipes

later versions of syslog have support for writing to named-pipes. a named-pipe is a special type of file that implements a simple fifo stream, allowing processes to talk to each other. we'll exploit named-pipes to implement real-time messaging between syslog and our mailer. in our example, we'll take all critical messages written to the local0 facility and (in addition to logging) send them to the mail recipient, fireman@example.com.

configuring syslog to write to a named-pipe

first, create a named-pipe for critical messages, for example:

# mkdir /etc/syslog.pipes
# mknod /etc/syslog.pipes/criticalMessages p
# chmod 600 /etc/syslog.pipes/criticalMessages

next, configure syslog to log all critical messages written to the local0 facility to this pipe. add the following statement to your syslog.conf file.

local0.crit   |/etc/syslog.pipes/criticalMessages

sending out messages

the final step is to mail out any messages that are written to the pipe. you can do this with a simple shell script. i've included an example below, let's call it /usr/bin/syslogMailer:

#!/bin/bash

# syslogMailer: a script to read stdin and turn each line into an alert
# email typically this is used to read a named-pipe written to by syslog
#
#   example usage: syslogMailer < /etc/syslog.pipes/criticalMessages
#

alertRecipient="fireman@example.com"      # the mail recipient for alerts
TMOUT=1                                   # don't wait > 1 second for input

# process each line of input and produce an alert email
while read line
do
   # remove any repeated messages
   echo ${line} | grep "message repeated" > /dev/null 2>&1
   if test $? -eq 1
   then
      # send the alert
      echo "${line}" | mailx -s "critical error on syslog" ${alertRecipient}
   fi
done

daemon vs cron?

you'll notice that i've included the following line in the script:

TMOUT=1                                 # don't wait > 1 second for input

this line specifies a one second timeout for the bash builtin, read. the script therefore runs to completion after processing one batch of zero or more messages. this allows you to schedule it in cron to run, say, every 5 minutes with a statement like:

# m h  dom mon dow   command
0-59/5 * * * * /usr/bin/syslogMailer < /etc/syslog.pipes/criticalMessages > /dev/null 2>&1

alternatively, if you'd like to turn this script into a log-running daemon that will sit in an endless loop and send out messages as soon as log statements arrive, remove the timeout line and surround the read statement with an while-true loop i.e.

# process each line of input and produce an error message
while :
do
   while read line
   do
      [...]
      # send the alert
      echo "${line}" | mailx -s "critical error on syslog" ${alertRecipient}
   done
done

the daemon approach is a little more efficient and sends out emails synchronously. it has the disadvantage that if your daemon terminates unexpectedly, alerts will stop until the daemon is restarted. the cron based implementation is arguably more robust in this regard. the cron approach also allows you to batch up notifications into n minute chunks. 5 minutes in our example cron file above.

4
  • Ended up basing my solution on this so I could daemonize the emailing. Thanks for the link.
    – jason
    Jun 26, 2009 at 22:46
  • This seems too much of a hack. Why would you spend time scripting when there is stable open source apps to do that for you? (in real time too). *see my reply.
    – sucuri
    Jun 29, 2009 at 2:07
  • 2
    Not a hack at all. It's the way this kind of thing often gets done. Jun 29, 2009 at 3:04
  • Link doesn't seem to work.
    – Yd Ahhrk
    Apr 15, 2021 at 20:25
7

Notifications via syslogd(8)

On my OpenBSD server, I log and email important messages from my web applications, which use facility local1. Here is my /etc/syslog.conf to make it happen:

local1.err    /var/log/example.com
local1.err    |while read log; do echo "$log" | /usr/bin/mail -s SYSLOG me@example.com; done

Notice that the while loop infinitely reads each line from syslogd and then pipes it to mail via echo. This is important. Once echo outputs its line it terminates the pipe, sending mail an EOF so it can email the log message.

In other words, you can't pipe directly to mail via syslogd like so:

local1.err    |/usr/bin/mail -s SYSLOG me@example.com

because syslogd will continue writing to the pipe until it is itself terminated or sent a HUP signal, at which time mail would send the entire set of log messages in one big email.

Notifications via newsyslog(8)

Scheduling newsyslog in cron is another way to get messages at a slower rate or in bulk.

For example, if you wanted a daily email digest of the log messages, set the M flag and specifying a monitor email address in /etc/newsyslog.conf:

# logfile_name        owner:group  mode  count  size  when  flags  monitor
/var/log/example.com  root:wheel   640   7      *     24    M      me@example.com

Then schedule newsyslog in crontab:

# minute hour  mday  month  wday  command
0        *     *     *      *     /usr/bin/newsyslog
1-59     *     *     *      *     /usr/bin/newsyslog -m

The -m option for newsyslog(8) states:

Monitoring mode; only entries marked with an `M' in flags are processed. For each log file being monitored, any log output since the last time newsyslog was run with the -m flag is mailed to the user listed in the monitor notification section.

4

You may want to look at logcheck or logwatch. Logcheck will email you hourly with log lines that don't match a set of patterns. I suspect you could probably make it do it more frequently. I don't know of any tools that do it by watching the logfiles, but I'm sure there's something out there that does it.

1
  • Yeah, I'd prefer not to monitor and parse log files after they are written, but just hook into the logging process so I can get instant notifications.
    – jason
    Jun 26, 2009 at 19:29
1

I would use OSSEC. It monitors your logs in real time and allows you to easily alert by email (or other means) when specific events are matched. Simple to use, scalable and open source.

link: http://www.ossec.net

1
  • 1
    This may be overkill for the application. However, the package does a lot and if there's a need then it may be a fit. Jun 29, 2009 at 3:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.