Munin : notification when everything is ok

Question

I use one Munin-Master to monitore 20+ servers, all run fine, except for one server. Last three Munin mails receveid :

05h25

infra :: backup2.infra :: Disk usage in percent OKs: /var is 22.55, /run/user/1001 is 0.00, /home is 8.87, /mnt/usb1 is 30.55, /export/oxa is 51.58, /tmp is 0.60, /dev/shm is 0.00, /space2 is 40.39, /run is 8.77, /run/lock is 0.00, /run/user/65534 is 0.00, /space is 76.38, /sys/fs/cgroup is 0.00, / is 18.46.

infra :: backup2.infra :: Inode usage in percent OKs: /dev/shm is 0.00, /run is 0.05, /space2 is 7.44, /run/user/65534 is 0.00, /run/lock is 0.00, /sys/fs/cgroup is 0.00, /space is 0.24, / is 8.07, /dev is 0.03, /home is 0.13, /mnt/usb1 is 0.51, /export/oxa is 0.01, /tmp is 0.02, /var is 2.02, /run/user/1001 is 0.00.

07h00

infra :: backup2.infra :: Inode usage in percent OKs: /home is 0.13, /var is 2.02, /run/user/1001 is 0.00, /dev/shm is 0.00, /run is 0.05, /run/lock is 0.00, /space is 0.24, /run/user/1003 is 0.00, /tmp is 0.02, / is 8.07, /space2 is 7.44, /mnt/usb1 is 0.51, /export/oxa is 0.01, /dev is 0.03, /sys/fs/cgroup is 0.00.

08h50

infra :: backup2.infra :: Inode usage in percent OKs: /run/user/1001 is 0.00, /tmp is 0.02, /dev is 0.03, /run/user/0 is 0.00, /dev/shm is 0.00, /run is 0.05, /space is 0.24, /sys/fs/cgroup is 0.00, /mnt/usb1 is 0.51, / is 8.07, /home is 0.13, /space2 is 7.44, /run/lock is 0.00, /var is 2.02, /export/oxa is 0.01.

infra :: backup2.infra :: Disk usage in percent OKs: / is 18.46, /mnt/usb1 is 30.62, /sys/fs/cgroup is 0.00, /export/oxa is 51.62, /run/lock is 0.00, /var is 22.29, /space2 is 40.39, /home is 8.87, /tmp is 0.60, /run/user/1001 is 0.00, /space is 76.49, /dev/shm is 0.00, /run is 9.27, /run/user/0 is 0.00.

All is Ok, no error in master logs, but still I received lots of these messages.

Here are the logs on master about this node

munin-update.log:2016/03/25 10:40:24 [WARNING] Service nfs4_client on backup2.infra/backup2.admin2:4949 returned no data for label fsinfo munin-update.log:2016/03/25 10:40:21 [WARNING] Service nfs_client on backup2.infra/backup2.admin2:4949 returned no data for label remove

munin-update.log:2016/03/25 09:55:06 [INFO] starting work in 29082 for backup2.infra/backup2.admin2:4949. munin-update.log:2016/03/25 09:55:06 [INFO] node backup2.infra advertised itself as backup2 instead. munin-update.log:2016/03/25 09:55:12 [INFO]: Munin-update finished for node infra;backup2.infra (6.67 sec) munin-update.log:2016/03/25 09:55:13 [INFO] Reaping Munin::Master::UpdateWorker. Exit value/signal: 0/0

Configuration for notification

 contact.devs.command mail -s "Munin notification ${var:host}" [email protected] 
 contact.devs.always_send warning critical

Here is the configuration file for this node (generated, as for all nodes)

[backup2.infra]
     address backup2.admin2
     use_node_name yes
     diskstats_latency.backup2_store_export.avgrdwait.warning :7
     diskstats_latency.backup2_store_export.avgwrwait.warning :7
     diskstats_latency.backup2_store_export.avgrdwait.critical :10
     diskstats_latency.backup2_store_export.avgwrwait.critical :10

Munin Master and node version: 2.0.25-1 (both Debian Jessie)

Where can I watch to undestand and resolve ?

Answer 1

The df plugin in Debian also checks the dynamically mounted filesystems under /run/user/<uid> which appear when a user logs in and which disappear when the user logs out. Even though all levels are OK, this appearance and disappearance is considered a change which triggers an email.

You should be able to avoid this by creating a file called /etc/munin/plugin-conf.d/df with the following content:

[df*]
env.exclude_re /run/user/

To check if your settings work and to list which paths the df plugin considers, use the following command:

munin-run -d df

If you are happy with the outcome, restart the munin-node service (service munin-node restart).

Answer 2

Recent Munin in Debian and derived distributions should handle this according to Debian bug #788736.

Some logic around tmpfs type mounts (which /run/user/* is) has been fixed in the Munin upstream project. As far as I can see, they are however not excluded pr. default (probably the Debian specific configuration that does this).

Answer 3

For me, docker volumes were another cause of this error.

I've ended up using this configuration to fix both the @Oliver's issue with /run/user as well as my docker one;

[df*]
env.exclude_re ^(/run/user/|/var/lib/docker)