how to "batch" output of "tail -F" or output all data in a stream every X seconds?

Question

I am trying to monitor a log file in real-time. Say I issue a command such as:

tail -F mysystem.log|grep -i error|mail ...

This is supposed to monitor my log file, every time it sees an error, email the line containing the error.

However, it is possible that something blows up and I get hundreds of errors a second. I don't want to kill my mail server as well by sending hundreds of emails a second. So I would like some sort of delay operator:

tail -F mysystem.log|grep -i error|window X|mail ...

This will hold all my error messages for X seconds, then release them together. This way, at most, I'll get an email every X seconds.

As a bonus, I'd love to be able to do the following:

tail -F mysystem.log|grep -i error|window X Y|mail ...

Same as the previous command, but if the number of lines in window is greater than Y, then send out an email containing Y messages (window clears and resets).

How can I do this without writing a PERL program? I'd like to stick to what is already built into UNIX.

Answer 1

This is solved quite nicely with rsyslog and the ommail (and omfile module if your log is not syslog compatible). I have multiple devices from different vendors in a secure data center all sending their syslog events to my central rsyslog server. Using the ommail module, I setup email alerts on various conditions; the built-in interval threshold ensures that I don't get flooded with messages (I set it to send a maximum of one alert per type every 60 minutes).

Works well, easy to setup: just make sure you get the latest version of rsyslog; Ubuntu 10.04.2LTS still ships with 4.2 which was released almost two years ago and is missing some bug fixes.

Answer 2

I think you can do this with named pipes.

First create the pipe and the mailing process:

mkfifo logwatch_fifo
while true; do cat logwatch_fifo | mail ...; sleep 10; done

Then do the monitoring and write to the fifo:

tail -F mysystem.log | grep -i error > logwatch_fifo

You can adjust it to your liking, basically doing whatever you want with the output buffered in the fifo using some variation of while loop and sleep.

You need the named pipe here instead of a regular pipe (just piping the output to a (while true ...) subshell, for example) because if you do the latter cat never lets go of the pipe if tail doesn't close its end.

Answer 3

You could give 'Tenshi' a try. Written in PERL :), capable to monitor multiple log files and triggered by reg-exp.

Debian package, Ubuntu package, Gentoo ebuild