1

I have some infrastructure (servers, switches etc) monitored by a Zabbix server, setup to alert in case of issues; so far so good. But what if the Zabbix server itself (or any of the underlying infra) experiences a problem?

One idea would be to publish some sort of heartbeat, which can be monitored by an external system. I'm thinking to use the Zabbix API (probably using py-zabbix) to expose this over http and have it monitored using smth like monitor.us.

Before I take the plunge, I can't help wondering if something simple already exists to cover this? Or is this even a good approach? Would monit be a better approach compared to a custom Python script? (not sure this passes the "simplicity" test)...

  • Note that Zabbix is able to monitor itself using internal checks, please see zabbix.com/documentation/2.4/manual/config/items/itemtypes/… . – asaveljevs Dec 6 '15 at 10:19
  • Fair enough but what if Zabbix (or the box on which it runs) crashes (I suppose we could have a 2nd server for HA) or if something goes wrong on the network and the Zabbix server becomes isolated? – sxc731 Dec 6 '15 at 10:23
  • If Zabbix server crashes, then, of course, it will not be able to monitor itself. For this reason, you should probably consider external monitoring, as your own question and the current answer suggest. Still, internal checks are a good way of monitoring Zabbix server health, as long as it is running and connected. – asaveljevs Dec 6 '15 at 14:00
1

So here's what I ended up doing:

  1. Wrote a fairly simple Python script which uses pyzabbix to interrogate Zabbix for the set of "triggers" currently failing (see snippet below). This runs periodically on a background Thread (so it has to be thread-safe).
  2. I used web.py to expose this to the external monitoring system.

There was an unexpected hiccup: the Zabbix API still responds even if the Zabbix server is down and there is no way to interrogate the status of the server - which was the main thing I wanted to monitor. Thankfully a patch exists to allow such server status queries.

Here is the code to query the set of failing Zabbix triggers (adapted from an example which comes with pyzabbix). If you need the code for the full monitor, please ask in a comment and I'll post it on github.

def __query_unacked_triggers(self):
    """ queries for currently tripped _triggers which haven't been acked """
    return self._zapi.trigger.get(
        only_true = 1,
        filter = { 'value': 1 },
        skipDependent = 1,
        monitored = 1,
        active = 1,
        output = 'extend',
        expandDescription = 1,
        expandData = 'host',
        withLastEventUnacknowledged = 1,
    )
0

External heart beats is the way to go.

I am not aware of any such tools, I have my own for the monitoring system I use. Just make sure that it is a heart beat and not a failure report. Eg. you always report that everything is ok and when the monitoring system does not receive your heartbeat for a longer period then it should let you know.

  • Thanks Florin; what about the approach to implement/expose those: write my own Python script of use smth existing? – sxc731 Dec 6 '15 at 9:10
  • I am not aware of any such tools, I have my own for the monitoring system I use. Just make sure that it is a heart beat and not a failure report. Eg. you always report that everything is ok and when the monitoring system does not receive your heartbeat for a longer period then it should let you know. I will add this to the answer as well. – Florin Asăvoaie Dec 6 '15 at 9:38
  • Here is what I have done.. Create a small AWS public cloud instance with a python script that sits there bound to a port. It accepts connections and then sets a last seen heartbeat.. If the last heartbeat is over 5 minutes old send a notification to pagerduty via their API – Mike Jan 19 '16 at 23:33

protected by Community Feb 22 '16 at 23:07

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.