What is the simplest way to monitor log files for (a) string/regex matches AND (b) lack of output?

Question

On a single Linux server, there are often lots of independent (perhaps interlinked) services outputting numerous logs to files at known paths.

Sometimes bad stuff happens and is written about, to the logs. Sometimes the services lock up and the logs get no (or perhaps very few) new entries.

What's the simplest way to monitor both of these kinds of events using one tool/service/system?

Answer 1

I don't know if this is "the simplest way", but I think this is the proper way: Use alerting/monitoring systems/services such as Nagios, Zabbix, Zenoss or any one of 100s other similar tools. They will monitor your server for specified parameters (process is running/active, port accepting connections, activity in the log file, content of log files) and perform actions based on rules(alert you, restart service, etc).

Answer 2

There is now a Nagios Log Monitoring plugin available. See the following link:

Nagios Log Monitoring

Read it carefully and see if that's what you want.

Your first question; monitoring logs for strings/regex:

From my experience with the logrobot tool used on that page, i know monitoring any type of log is not an issue, regardless of the log format. the example used on that page should affirm this.

Your second question; monitor log to make sure there's output:

now to also monitor and alert if there's any output, that is possible as well. i'm trying to think off the top of my head how to do that. but try this:

logrobot autonda /log/file/path 60m '.' '.' 2 2 maxclient -ndnotfoundn

logrobot = name of the tool

autonda = the feature to be used for your scenario

/log/file/path = path to your log file

60m = if the last time the file was last modified is over 60m, the tool wont go any further.

'.' = this is searching for any thing in the log file

'.' = this is searching for any thing in the log file

2 = if at least two lines aren't found, alert

maxclient = the name you're giving to this session. you can call this anything you want.

-ndnotfound = this is the option you're passing to the tool

you can of course change the parameters to fit your specific needs.

Answer 3

SNMP (Nagios Net-Snmp) is very good but painful to set up correctly. If you want minimal cost consider using a cron job - you write a script in bash, cron submits it periodically.

Inactivity:

create a file, ex: list.txt, of directories and logfilenames like this, these are pretend samples. Columns are path logfilename max_hours_inactive:

/var/log  syslog     9
/var/adm  message    12

#!/bin/bash
# inactive.sh
while read path fname mxhrs
do
   oops=$( /usr/bin/find $path -mtime +${mxhrs} -name $fname)
   [ $#oops -gt 0 ]  &&  echo "$fname inactive for $mxhrs" |/usr/bin/mailx -s "$fname warning"
done < /path/to/list.txt

Create another file, list1.txt, with columns file file spec and regex EX:

/path/to/mylfile.log   "(Warning|Fatal)"

#!/bin/bash
# error.sh
while read path regex
do
   /usr/bin/egrep -q "$regex" $path  &&
        echo "$path has error notification" |/usr/bin/mailx -s "$path warning"
done < /path/to/list1.txt

Use crontab-e to enter when you want the jobs to run ex: every 10 minutes on workdays, every hour on weekends

0,10,20,30,40,50 * * * 1-5 /path/to/error.sh >> /tmp/error.log.$(date +%Y%m%d) 2&>1
0 * * * 0,6  /path/to/error.sh >> /tmp/error.log.$(date +%Y%m%d) 2&>1
0,10,20,30,40,50 * * * 1-5 /path/to/inactive.sh >> /tmp/error.log.$(date +%Y%m%d) 2&>1
0 * * * 0,6  /path/to/inactive.sh >> /tmp/error.log.$(date +%Y%m%d) 2&>1