I have a couple (say 40) of workstations and servers that are configured using Ansible in the pull configuration. Each machine has a cron job that uses
ansible-pull to pull a git repository and apply it onto the host itself. The results are currently emailed to my account.
In the emails I have to check whether some of the hosts have not send an email on that day to figure out that they are either just turned off or have some other issue. If a machine is missing I try to connect via SSH. If it is turned off, that's fine. Otherwise I try to fix the issue.
A couple of days ago I set up Nagios on one server to get an overview of the machines. Setting up monitoring via ping and SSH was easy enough, now I see which machines are up and which are down.
Now I would like to somehow also monitor the Ansible state in Nagios. I did not found a direct plugin on the Nagios Exchange, so I had the following ideas:
Write the output of
ansible-pullinto a log file with a timestamp and then let the Nagios host retrieve all the logs from the machines. Then it would need to check that the last timestamp is not older than 24+X hours, where X is some margin like 1 hour.
Convert everything back to the push architecture and just run
ansible-playbookon the infrastructure server. This then creates some file and I have an overview there, which might be somehow parsable by Nagios.
Port our configuration management to Puppet as that seems to have some monitoring build in and use that instead of Nagios.
Buying Ansible tower might be a solution, but there won't be money for it. Until now people just have come to my office if their workstation hangs, so we do not really have a business need for a monitoring system. I want to do it with free software only.
The first option seems to be the easiest from the current setup, I am just not sure what a sensible implementation would be. An answer would be greatly appreciated if it provides me with a rough outline for the implementation and pointers to needed plugins for Nagios.