0

I have Logstash version 1.4.2 analysing haproxy logs using the HAPROXYHTTP pattern, and it is already sending counters and timing information to statsd for aggregation and onward storage in Graphite/Whisper.

Now I would like to count the specific numbers of times that a particular path element is contained within the URI component, then send this value to statsd as well.

The path that I am interested in would be: /important/new

I have tried adding a tag using a mutate and a conditional regex, but the results have so far shown that more of the logs match than I had intended. I think that I'm failing to understand something about how the filter section is supposed to work.

My logstash configuration attempt currently looks like this, although I have removed some of the working metrics for brevity.

input {
  file {
    type => "haproxy"
      path => "/var/log/haproxy/haproxy.log"
  }
}

filter {
  if [type] == "haproxy" {
    grok { 
      match => { "message" => "%{HAPROXYHTTP}" }
    }
    if [http_request] =~ /^\/important\/new$/ {
        mutate { add_tag => "important" }
    }
  }
}

output {
  if  [type] == "haproxy" {
    statsd {
      host => "statsd-host"
        count => [
          "haproxy.%{frontend_name}.%{backend_name}.%{server_name}.response_size", "%{bytes_read}"
        ]
        increment => [
          "haproxy.%{frontend_name}.%{backend_name}.%{server_name}.hits",
          "haproxy.%{important}"
        ]
        timing => [
          "haproxy.%{frontend_name}.%{backend_name}.%{server_name}.response_time", "%{time_duration}",
        ]
     }
  }
}

Many thanks.

0

I believe that I have found out how to do this, so I thought I'd post my own answer.

The key part was to make the output section conditional on the tag as well. For example:

  output {
    if  [type] == "haproxy" {
      statsd {
        host => "statsd-host"
        count => [
          "haproxy.%{frontend_name}.%{backend_name}.%{server_name}.response_size", "%{bytes_read}"
        ]
        increment => [
          "haproxy.%{frontend_name}.%{backend_name}.%{server_name}.hits",
          "haproxy.%{frontend_name}.%{backend_name}.%{server_name}.responses.%{http_status_code}"
        ]
        timing => [
          "haproxy.%{frontend_name}.%{backend_name}.%{server_name}.response_time", "%{time_duration}",
          "haproxy.%{frontend_name}.%{backend_name}.%{server_name}.queue_time", "%{time_queue}",
           <snip snip more of these>
          "haproxy.%{frontend_name}.%{backend_name}.%{server_name}.response_size", "%{bytes_read}"
        ]
      }

      if "important" in [tags] {
        statsd {
          host => "statsd-host"
          increment => [
            "haproxy.important"
          ]
        }
      }
    }
  }

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.