Sensu alternative (?) where alarm thresholds defined on server (not monitored client)

Question

Question/TLDR;

Is there a Sensu -alternative (i.e operating system monitoring agent/server based on RabbitMQ) that defines its alarm thresholds on the central monitoring server and not on the monitored client server (as Sensu and Nagios do)?

RabbitMQ is required so no Zabbix et al, I'm afraid.

Background:

I have a large environments (Windows and RHEL) where I can't install orchestration tools (Puppet et al) and the amount of installed services should therefore be kept to a minimum.

I'm researching if I could develop a single agent that collects system information, relays logs (to Logstash) and reports on resource consumption. It would push all these values in to RabbitMQ and then Logstash could subscribe to the logs, a monitoring service could subscribe to the resource metrics (and create alarms from them), a CMDB system could subscribe to the system information etc.

However, I would want to just receive the information about resource consumption and create the alarms on the monitoring server and not have to change the thresholds on each server to change the alarm threshold.

I can't be the only person to find an agent like that useful...

Clarification:

If a server under Sensu monitoring is running out of disk, the Sensu agent checks the disk space, compares it against the CRITICAL threshold that's defined on that server and if the threshold is passed, a CRITICAL alarm is sent through RabbitMQ to the central monitoring server. To change the threshold without Puppet or somesuch, logging in to the server is required (right?)

The way I'd like this to work is that when a monitoring agent checks its disk space, it just sends the amount of available disk (or used disk and total etc) through RabbitMQ to the central server which then compares that value against the threshold defined on the central server and, if necessary, sends an alarm.

If the threshold needs to be changed, it's changed on the central server or multiple values from multiple servers can be compared to create an alarm.

This is kinda my main issue with Sensu, although I understand the decision to go with Nagios compatibility.

It would also be preferable if no central server -> monitored server traffic would be required. I suppose a kludge could be made where the central server sends the thresholds to the agent which then runs them as "local". The network for the environment makes this exceptionally tricky.

Thanks for any ideas anyone might have!

Answer 1

Using open source components, I'd use the following components (if you indeed do need to send metrics via RabbitMQ):

1. use collectd on the client side to send metrics into RabbitMQ with its AMQP plugin
2. consume the messages from RabbitMQ using graphite-amqp-tools and send them into Graphite

Now you have the metrics in Graphite, you can query it for your resource consumption. In my $WORK's environment, we have checks which query Graphite, with the alerting thresholds set on Nagios server. But now that you have Graphite (is has a http interface for querying which can return graphs, json, csv & plain text results) you could build/use anything as long as it can query Graphite.

Answer 2

Threshold can be defined on the sensu monitoring server side, see page 9 of http://samples.leanpub.com/sensumonitoringandmetrics-sample.pdf (but make sure safe_mode=false, if client definiton on server doesnt match exactly the one on the client, see page 12)

Answer 3

If I understand that correctly, you just need to push data to RabbitMQ. Isn't the topic then about how to push reported data into RabbitMQ?

Maybe you could use couple of options:

• agentless monitoring - you could use old (good) snmp and parse its data on a server. There are many tools which could do it and then you could forward it to wherever you want.

• agent-based monitoring - you could use for example cfengine which can check promises (pulled from central server) or which can just report the state of your filesystem and then you can parse its data.

Answer 4

Yes, this is exactly what dataloop.io is doing. Like Sensu, uses queue based architecture for real-time metrics and simple configuration, but the alerts can be configured server side not just in scripts like Nagios/Sensu.

Sensu also requires you to install other stuff like Graphite to get graphs too, so isn't very easy to setup.