I'm fairly new to monitoring and am currently trying to setup a monitoring solution for our network infrastructure. Our hardware is pretty old and can therefore fail at any time. We have taken precautions for this case (avoiding single points of failure wherever possible) but we want to at least be informed when something dies. A single machine monitoring our infrastructure is not suitable, as this machine can then fail without us knowing. I've thought of a few solutions to this problem of which none really satisfy me.
I have the following requirements:
- A single monitoring machine may fail at any time without causing problems
- Configuration can be done centralized (e.g. adding new host to be monitored)
- General ease of configuration and maintenance (I do not know who will be managing the network in a few years).
My ideas so far:
- Setting up OMD on two machines, one as a master the other as a replica. The problem I encountered here is that I don't seem to be able to tell both machines to monitor all hosts. I can only select one of the two monitor instances for each host.
- Setting up a "master" OMD that monitors everything and a "slave" that only monitors the master. I don't really like this solution because if the master dies at the same time as some other host, I will only know about the failure of the monitor.
- Configuring an OMD instance and exporting this configuration to another instance => mainainability not very good.
We do not have a lot of machines to monitor, there will probably always be less than 20 at any given time for the forseeable future. If you have any ideas how to achieve this, I'd be happy to hear them.