How can I debug why Munin won't talk to Nagios?

Question

I've followed the instructions to connect Munin to Nagios. NSCA is installed, echo -e "foo.example.com\ttest\t0\t0" | /usr/sbin/send_nsca -H localhost -c /etc/send_nsca.cfg works when run as the munin user, and I've added contacts nagios and the contact.nagios.command /usr/bin/send_nsca -H your.nagios-host.here -c /etc/send_nsca.cfg command to the Munin configuration.

But I don't get any NSCA messages in the logs. Not for failing to send messages on the Munin side, or receiving messages on the Nagios side. Here's my questions:

1. How can I trigger alerts for Munin to be able to test?
2. How can I determine what step is failing?
3. What should I be seeing in the logs if Munin fails to send NSCA messages?
4. What should I be seeing in the Nagios logs if Munin succeeds in sending messages?

Answer 1

It is not very clear from the instructions, but NSCA is only necessary if Nagios is not running on the same machine as Munin, and I see your NSCA is sending to localhost. This makes things much easier to setup and debug.

1) How can I trigger alerts for Munin to be able to test?

No idea, but will investigate and update the answer if I discover a quick method.

2) How can I determine what step is failing?

First, make sure you have the following lines in nagios.cfg:

accept_passive_service_checks=1
check_external_commands=1
command_file=/var/lib/nagios3/rw/nagios.cmd

To further debug Nagios issues, enable the debug log in nagios.cfg:

debug_level=256
debug_verbosity=2

Restart the Nagios service and keep an eye on the following logs:

/var/log/nagios3/nagios.*
/var/log/munin/munin-*

You can manually send passive notification to a specific host and service via:

sudo -u nagios echo "[$(date +%s)] PROCESS_SERVICE_CHECK_RESULT;$host;$service;$result;$notes" >> $command_file

More info here.