Logstash conditional output to statsd - based on haproxy request URI

Question

I have Logstash version 1.4.2 analysing haproxy logs using the HAPROXYHTTP pattern, and it is already sending counters and timing information to statsd for aggregation and onward storage in Graphite/Whisper.

Now I would like to count the specific numbers of times that a particular path element is contained within the URI component, then send this value to statsd as well.

The path that I am interested in would be: /important/new

I have tried adding a tag using a mutate and a conditional regex, but the results have so far shown that more of the logs match than I had intended. I think that I'm failing to understand something about how the filter section is supposed to work.

My logstash configuration attempt currently looks like this, although I have removed some of the working metrics for brevity.

input {
  file {
    type => "haproxy"
      path => "/var/log/haproxy/haproxy.log"
  }
}

filter {
  if [type] == "haproxy" {
    grok { 
      match => { "message" => "%{HAPROXYHTTP}" }
    }
    if [http_request] =~ /^\/important\/new$/ {
        mutate { add_tag => "important" }
    }
  }
}

output {
  if  [type] == "haproxy" {
    statsd {
      host => "statsd-host"
        count => [
          "haproxy.%{frontend_name}.%{backend_name}.%{server_name}.response_size", "%{bytes_read}"
        ]
        increment => [
          "haproxy.%{frontend_name}.%{backend_name}.%{server_name}.hits",
          "haproxy.%{important}"
        ]
        timing => [
          "haproxy.%{frontend_name}.%{backend_name}.%{server_name}.response_time", "%{time_duration}",
        ]
     }
  }
}

Many thanks.

Answer 1

I believe that I have found out how to do this, so I thought I'd post my own answer.

The key part was to make the output section conditional on the tag as well. For example:

  output {
    if  [type] == "haproxy" {
      statsd {
        host => "statsd-host"
        count => [
          "haproxy.%{frontend_name}.%{backend_name}.%{server_name}.response_size", "%{bytes_read}"
        ]
        increment => [
          "haproxy.%{frontend_name}.%{backend_name}.%{server_name}.hits",
          "haproxy.%{frontend_name}.%{backend_name}.%{server_name}.responses.%{http_status_code}"
        ]
        timing => [
          "haproxy.%{frontend_name}.%{backend_name}.%{server_name}.response_time", "%{time_duration}",
          "haproxy.%{frontend_name}.%{backend_name}.%{server_name}.queue_time", "%{time_queue}",
           <snip snip more of these>
          "haproxy.%{frontend_name}.%{backend_name}.%{server_name}.response_size", "%{bytes_read}"
        ]
      }

      if "important" in [tags] {
        statsd {
          host => "statsd-host"
          increment => [
            "haproxy.important"
          ]
        }
      }
    }
  }