Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Ok, a little back story first,

We have a Tomcat instance that connects to multiple service providers as a sort of "gateway." This Tomcat instance will generate errors if it cannot connect to a given provider. These errors are stored in a log file.

The problem is that we need to monitor the file for more than just a given string, we need to use several regular expressions to filter through all the meaningless stuff so that we aren't waking up at 3am because a user forgot his password. ;)

I already tried creating a python script that would sit between Zabbix and Tomcat, but with log files (that are rotated daily) that are in excess of 200MB, it's too CPU intensive (100% for 15+ seconds). I tried adding a "sleep(0.000005)" to slow it down a bit, but now it takes to long (> 1min) and still uses 25% CPU...

All in all, the solution is overshadowing the problem and I wanted to reach out to you guys and see if you know of a better solution, or anything else that might help.

share|improve this question
add comment

3 Answers 3

(1)Zabbix can handle regular expression by itself. You can monitor logs using regex with zabbix agent.

https://www.zabbix.com/documentation/1.8/manual/log_file_monitoring

(2)Using 'nice' command enables you to lower the CPU usage priority of the script.

http://linux.die.net/man/1/nice

Other than that, the more detailed information will be needed. * machine spec (CPU, mem, storage device, etc) * source of the python script

share|improve this answer
add comment
up vote 0 down vote accepted

Well, in the end we went with an overcomplicated Python script to do the job until something more viable comes along.

Just to give you an idea of how complicated this "script" is, here are the MINIMUM files/folder that must exist for it to work:

/usr/local/bin/tomcat_log_alert.py (755)
/usr/local/bin/tomcat_ack_errors.py (755)
/etc/${COMPANY}/tomcat-log-alert.conf (644)
/etc/cron.d/tomcat-log-alert (644)
/var/log/${COMPANY}/tomcat_errors/ (755)
/var/log/${COMPANY}/tomcat_errors_ack/ (755)
/etc/zabbix/zabbix_agentd.d/tomcat-log-alert.conf (644)

I really do not look forward to being the sole maintainer for this fracking band-aid.

share|improve this answer
add comment

The following is worth giving a look:

logrobot autonda /prod/tomcat/error.log 60m 'tomcat errors' 'password forgotten|failed login' 1 2 -show

The above command will search the log file for any lines that contain "tomcat errors". And from the lines that it finds, it'll remove any lines containing:

  • password forgotten
  • failed login

The above is only searching ONE log file for the defined entries. What happens if you want to search multiple log files for entries?

You can try something like this:

logrobot automul /prod/tomcat/error.log_MULTIPLE 60m 'tomcat errors' 'password forgotten|failed login' 1 2 -show

I may be a little off so you should check the either one of these sites to get the exact usage:

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.