Icinga 1.8.4 not displaying all the services it manages

Question

I'm new to managing Icinga. One of our Icinga 1.8.4 installs is not showing me all the services/hosts it manages. I've checked its status.dat file and it references the services and hosts I expect, but they're not showing up in the web interface... has anyone seen this before? And any suggestions about where to start debugging it?

The upper left corner says 7 UP but if I click on it only 6 show up. Further, it says 81 UP re: services, but only 55 shows up if I click on that number. Of those services it also says 1 / 0 / 0 CRITICAL but when clicking on that none are in the list. Yet, it keeps sending email about that service being critical, and it is referenced in the status file.

Answer 1

Answering my own question because I finally managed to fix the problem. Carefully reading through the config file I noticed this comment above its object_cache_file setting:

This option determines where object definitions are cached when Icinga starts/restarts. The CGIs read object definitions from this cache file (rather than looking at the object config files directly) in order to prevent inconsistencies that can occur when the config files are modified after Icinga starts.

On this machine the user & group ownership of Icinga's objects.cache were set such that icinga could read it, but not update it. So although it would keep state and report on a wider range of services, the web UI was frozen in time since 2014...