icinga2 checks pending forever

Question

I have a simple, top-down Icinga2 configuration with one master and one slave. There are two zones.

1. 'core' is the master zone.
2. 'plwg' lives on a slave endpoint.

Most (but not all) of my newly added checks remain in a pending state.

The health of the cluster is good, according to the Cluster Health Check. I see no errors in /var/log/icinga2/icinga2.log on either machine.

Where do I begin to troubleshoot this?

I just ran

icinga2 object list --type Endpoint

Object 'mon-prod.polywog.net' of type 'Endpoint':
  % declared in '/etc/icinga2/zones.conf', lines 6:1-6:24
  * __name = "mon-prod.polywog.net"
  * host = ""
  * log_duration = 86400
  * name = "mon-prod.polywog.net"
  * package = "_etc"
  * port = "5665"
  * templates = [ "mon-prod.polywog.net" ]
    % = modified in '/etc/icinga2/zones.conf', lines 6:1-6:24
  * type = "Endpoint"
  * zone = ""

Object 'plwg.polywog.net' of type 'Endpoint':
  % declared in '/etc/icinga2/zones.conf', lines 17:1-17:45
  * __name = "plwg.polywog.net"
  * host = "plwg.polywog.net"
    % = modified in '/etc/icinga2/zones.conf', lines 18:3-18:38
  * log_duration = 86400
  * name = "plwg.polywog.net"
  * package = "_etc"
  * port = "5665"
  * templates = [ "plwg.polywog.net" ]
    % = modified in '/etc/icinga2/zones.conf', lines 17:1-17:45
  * type = "Endpoint"
  * zone = ""

So I'm wondering why the zone is empty. zone = "" Why?

Answer 1

Oy vey.

I had two instances of services.conf on the master. One was in /etc/icinga2/conf.d and the other was in /etc/icinga2/zones.d/global-templates.

So I mv /etc/icinga2/conf.d/services.conf /etc/icinga2/conf.d/services.flonk and restarted Icinga2 on the master and voila. Problem solved. I have nothing in pending!