0

I am using rsyslog for sending data from nginx to kafka. Pipeline looks like following:

nginx -> unix_socket -> rsyslog -> omkafka module -> kafka

Omkafka module for rsyslog uses tcp for transferring messages, so I can easily count bytes of dataflow sent to kafka and number of tcp packets, but these metrics are not enough representative for me. Seems that I should use some application level solution for precise counting of messages.

I know that 'impstats' module able to count rsyslog messages, but it doesn't support omkafka module.

Looking for any advice to grab statistics for my case. Thanks!

1 Answer 1

1

Ok. finally impstat looks like a good solution.

With simple config it produces enough information, since omkafka by default adds its counters into global counters.

Config:

module(
  load="impstats"
  interval="10"             # how often to generate stats
  resetCounters="on"        # to get deltas (e.g. # of messages submitted in the last 10 seconds)
  log.file="/var/log/impstats"     # file to write those stats to
  log.syslog="off"          # don't send stats through the normal processing pipeline. More on that in a bit
)

Output for my pipeline:

Tue Oct  4 12:40:22 2016: imuxsock: origin=imuxsock submitted=12942 ratelimit.discarded=0 ratelimit.numratelimiters=0 
Tue Oct  4 12:40:22 2016: omkafka: submitted=12942 maxoutqsize=1434 failures=0 topicdynacache.skipped=0 topicdynacache.miss=0 topicdynacache.evicted=0 

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .