2

I'm using JMX to monitor an Apache Kafka cluster with collectd's GenericJMX plugin. However, I'm not sure what some of the MBean parameters mean.

Particularly, I have one metric called kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec, showing the number of messages per second received by one machine. This is my configuration file for this MBean:

<MBean "kafka-all-messages">
  ObjectName "kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec"
  InstancePrefix "all"
  <Value>
    InstancePrefix "kafka-messages-in"
    Type "counter"
    Table false
    Attribute "MeanRate"
  </Value>
</MBean>

I don't know which Type is the correct one for this measurement, such as gauge, counter or another one. I've tried reading this documentation on collectd by couldn't really understand where I should apply each measurement type.

Could someone explain me it?

2 Answers 2

2

This looks like a gauge, by virtue of being a 'Per' value -- Kafka is already normalizing this into a particular time frame. If the name was just 'MessagesIn' I'd expect it to be a counter -- a number that continually increases as messages come in.

In other words, if the bean returns the 'right now' of something, it is a gauge. If it returns the 'total so far' of something, it is a counter. MessagesInPerSec is a 'current state' value so is a gauge.

1

I finally manage to get it working. Pretty much all Kafka metrics in JMX have multiple attributes. For instance, this kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec contains the following attributes:

  • Count
  • EventType
  • FifteenMinuteRate
  • FiveMinuteRate
  • MeanRate
  • OneMinuteRate
  • RateUnit

When selecting the Type, you must choose whether you want it to retrieve the value as is or its difference for the interval collectd is configured to report metrics. Hence, gauge is the value and counter is the difference.

The most flexible way to report Kafka's JMX metrics is to simply use the Count attribute, a monotonically increasing counter that is incremented whenever a new message arrives, and report it as a counter. If collectd is configured to report every 60 seconds, this will be the number of messages coming in per minute. If it is reporting every 10 minutes, the value will be the number of messages per 10 minutes.

<MBean "kafka-all-messages">
  ObjectName "kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec"
  InstancePrefix "all"
  <Value>
    InstancePrefix "kafka-messages-in"
    Type "counter"
    Table false
    Attribute "Count"
  </Value>
</MBean>

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .