4

I am trying to setup email alerts using logstash. Right now it emails me EVERY time the pattern "Error" is parsed into my log file which can lead to a lot of unnecessary emails. I'd like to create a conditional rule so that let's say "X logfile has the pattern Error 3x in 1 minute email me". This way I don't get overwhelmed with emails.

Here is my current config:

input {
  file {
#    sincedb_path => /path/to/whatever/
    path => "/opt/test.log"
    type => "test_log"
  }
}

filter {
   dns {
      add_field => [ "IPs", "Logs, from %{host}" ]
      type => [ "MESSAGES" ]
      resolve => [ "host" ]
      action => [ "append" ]
     }
}

filter {
  if [message] == "Error" or [message] == "error" {
    throttle {
      before_count => 1
      after_count => 3
      period => 10
      key => "%{message}"
      add_tag => "throttled"
  }
} }

output {
#  stdout { codec => rubydebug }
   redis { host => "redis_IP" data_type => "list" key => "logstash" }
   if "throttled" not in [tags] {
      email {
        from => "[email protected]"
        to => "[email protected]"
        subject => "Alert from  %{path}, from %{host}"
        body => "Message is: ]\n'%{message}'. \nLog file:\n %{path}:\n\n%{message}.\n More information can be viewed in Kibana"
        }
    }
}
10
  • 'period' is in seconds. You're reseting the count every 10 seconds. Sep 8, 2014 at 16:54
  • I'm doing that to test. I want to only trigger an email if "error" is parsed into /this/test.log 3x in 15 seconds (or whatever). I have another config that works but emails me EVERY time "error" is parsed which is leading to way too many emails.
    – Gabriel
    Sep 8, 2014 at 16:58
  • So what is this config doing that you don't want it to? Sep 8, 2014 at 17:06
  • on pastebin: pastebin.com/LGsB3qQS
    – Gabriel
    Sep 8, 2014 at 17:08
  • That's a different config than the one in this question, and contains no throttle. Sep 8, 2014 at 17:12

1 Answer 1

7

We just setup Riemann to handle alerting based on log messages.

Riemann can read a stream of log messages from logstash and send out alerts based on the contents.

One of the advantages with riemann is you can rollup all messages from a certain time into one email. This way you will not get to many e-mails but you will still get all your messages.

Much more examples can be found at http://riemann.io/howto.html

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .