Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

With logs getting captured in syslog-ng, I'd like to be able to automatically monitor the logs and receive an alert if NO log events appear that match a certain criteria. For instance, for a subscription-based website, if 6 hours elapse with no orders, then email or text this person or group of people.

What is a good way to do that?

share|improve this question
    
See this question. serverfault.com/questions/101744/…. See the examples there and extend them to search for and count matches within a specific time range. –  Zoredache Oct 11 '12 at 22:43
    
you may also want to look into turning on MARK messages, to ensure the syslog is actually working, as well. –  Sirex Oct 11 '12 at 22:44

8 Answers 8

up vote 2 down vote accepted

http://labs.consol.de/nagios/check_logfiles is a Nagios plugin which is used to monitor logfiles. Usually you check, if there is a certain pattern (error message). But it's alos possible to reverse this. For example, if you run check_logfiles --logfile /var/log/mybackup.log --criticalpattern '!backup succeeded' every morning, you will get an alert if there was no 'backup succeeded' message entry since the last run of check_logfiles.

Gerhard

share|improve this answer

As the information that a certain event should occur at least every six hours is specific to the application writing to the log, it would be best if it could monitor itself and write a log entry if the event does not happen as it should. If the application writing log information is developed locally I would recommend this solution.

If that is not possible, I would keep the mechanism which makes sure the event has occurred as close to the application as possible. Maybe a watchdog could be started in the background from the same start script that starts the application performing logs. When the application is stopped, the script also stops the watchdog.

Another option, in case you would like to keep the start script in its original condition for some reason, would be to create a cron-job which performs the log watching.

In either case. Make sure the level of the log entry is severe enough that you are alerted to take a look at it using whatever tool you are using to monitor logs. It's better, more future proof and inclusive for things you didn't think of but should monitor, then to listen for a log level than a specific log entry. This means at least WARNING level.

share|improve this answer

Zabbix is another monitoring solution similar to Nagios. Zabbix has the ability to monitor files for various strings and initiate a triggered alert based off of the criteria you specify (found or not found). Zabbix also supports "looking" back or from a various point in time in the log which helps prevent older events from being false positives. Alerts can be configured to send emails or SMS.

Zabbix Website: http://www.zabbix.com/

Some of Zabbix's Log Monitoring Docs: http://www.zabbix.com/documentation/2.0/manual/config/items/itemtypes/log_items

share|improve this answer
    
Does this do what the question asks for - send an alert on the absence of an event? Because your answer doesn't include that information. –  mfinni Oct 24 '12 at 18:29
    
Sorry, I clarified a bit more that Zabbix can look for a string or the absence of a string specified by the user's configuration for the Item/Trigger. –  bmurtagh Oct 24 '12 at 18:55
    
Thanks, that makes this a much better answer! –  mfinni Oct 24 '12 at 19:59
    
To add to my post, you can also create wrappers (scripts) and define what Zabbix refers to as UserParameters that are custom items. I have a few in place across various servers to monitor non-traditional processes or tasks. A wrapper could be written that executes and outputs a result to a log that is then monitored by Zabbix and alerts accordingly. –  bmurtagh Oct 25 '12 at 13:41

You could use Shell/Perl Scripts to do this task. You can run the Shell / Perl scripts through Cron.

Nagios would be a better solution. It's an open source tool, so no need to pay anything, but need to fight with it for configuring it. ;)

If you want a commercial product, Splunk is known for the logfile analysis.

share|improve this answer
    
If you need to "fight" with the configuration it means you have failed to read the documentation. –  John Gardeniers Oct 19 '12 at 1:56

The way I've approached this in the past is by forwarding syslog traffic to a database, and then just querying the database for the last received log message from a particular hostname, with it's timestamp.

Which is just one of the positive benefits from logging to a database.

share|improve this answer

I believe the easiest way to take care of this is by writing very simple script in language of your choice. You just need to grep through the syslog for a specific messages and send email if none found. Including some sanity checks I can't imagine more than 20 lines. Stick this is cron and you are all set. The drawback is that by using this method, the cron job has to run as root in order to read the logs.

What you could also do is to take a look at LogAnalyzer. Unfortunately, I don't remember if it is capable of sending emails. Nevertheless, it uses MySQL and stores the log info there. With LogAnalyzer setup, you would have two copies of your logs, as they go in parallel - one in syslog, one in the DB. If you have an user for this DB, capable of reading its contents, you could avoid running the above mentioned cron job. It may not be that much better but could add another protection layer.

share|improve this answer

We have done something similar where we get alerted if we don't get a new order on the website within the last N hours. We created a PushMon URL with a schedule of "every 3 hours". We then called the PushMon URL in our thank you page (JavaScript). If we don't get an order, PushMon will notice the missing call, and will send an alert.

PushMon works well for alerting you about events that do not occur.

share|improve this answer

Nagios can do this for you with ease. Personally, i like any thing that i can easily run on the commandline and/or set up easily via Nagios or Crontab.

LOGROBOT can do exactly what the OP is asking for...in one simple command line entry:

./logrobot autonda /prod/apps/mylogfile 60m 'orders submitted' '.' 1 2 subscription_orders -ndnotfoundn

Just change the bolded strings and of course the log file path to match your needs.

Basically, this command will alert as warning if at least 1 'orders submitted' entry is NOT found in the log file /prod/apps/mylogfile. It will alert as critical if at least 2 'orders submitted' entries are not found in the log.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.