4

I am trying to setup email alerts using logstash. Right now it emails me EVERY time the pattern "Error" is parsed into my log file which can lead to a lot of unnecessary emails. I'd like to create a conditional rule so that let's say "X logfile has the pattern Error 3x in 1 minute email me". This way I don't get overwhelmed with emails.

Here is my current config:

input {
  file {
#    sincedb_path => /path/to/whatever/
    path => "/opt/test.log"
    type => "test_log"
  }
}

filter {
   dns {
      add_field => [ "IPs", "Logs, from %{host}" ]
      type => [ "MESSAGES" ]
      resolve => [ "host" ]
      action => [ "append" ]
     }
}

filter {
  if [message] == "Error" or [message] == "error" {
    throttle {
      before_count => 1
      after_count => 3
      period => 10
      key => "%{message}"
      add_tag => "throttled"
  }
} }

output {
#  stdout { codec => rubydebug }
   redis { host => "redis_IP" data_type => "list" key => "logstash" }
   if "throttled" not in [tags] {
      email {
        from => "logstash@shipper.com"
        to => "sysadmin@something.com"
        subject => "Alert from  %{path}, from %{host}"
        body => "Message is: ]\n'%{message}'. \nLog file:\n %{path}:\n\n%{message}.\n More information can be viewed in Kibana"
        }
    }
}
  • 'period' is in seconds. You're reseting the count every 10 seconds. – Alain Collins Sep 8 '14 at 16:54
  • I'm doing that to test. I want to only trigger an email if "error" is parsed into /this/test.log 3x in 15 seconds (or whatever). I have another config that works but emails me EVERY time "error" is parsed which is leading to way too many emails. – Gabriel Sep 8 '14 at 16:58
  • So what is this config doing that you don't want it to? – Alain Collins Sep 8 '14 at 17:06
  • on pastebin: pastebin.com/LGsB3qQS – Gabriel Sep 8 '14 at 17:08
  • That's a different config than the one in this question, and contains no throttle. – Alain Collins Sep 8 '14 at 17:12
7

We just setup Riemann to handle alerting based on log messages.

Riemann can read a stream of log messages from logstash and send out alerts based on the contents.

One of the advantages with riemann is you can rollup all messages from a certain time into one email. This way you will not get to many e-mails but you will still get all your messages.

Much more examples can be found at http://riemann.io/howto.html

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.