4

I use one Munin-Master to monitore 20+ servers, all run fine, except for one server. Last three Munin mails receveid :

05h25

infra :: backup2.infra :: Disk usage in percent OKs: /var is 22.55, /run/user/1001 is 0.00, /home is 8.87, /mnt/usb1 is 30.55, /export/oxa is 51.58, /tmp is 0.60, /dev/shm is 0.00, /space2 is 40.39, /run is 8.77, /run/lock is 0.00, /run/user/65534 is 0.00, /space is 76.38, /sys/fs/cgroup is 0.00, / is 18.46.

infra :: backup2.infra :: Inode usage in percent OKs: /dev/shm is 0.00, /run is 0.05, /space2 is 7.44, /run/user/65534 is 0.00, /run/lock is 0.00, /sys/fs/cgroup is 0.00, /space is 0.24, / is 8.07, /dev is 0.03, /home is 0.13, /mnt/usb1 is 0.51, /export/oxa is 0.01, /tmp is 0.02, /var is 2.02, /run/user/1001 is 0.00.

07h00

infra :: backup2.infra :: Inode usage in percent OKs: /home is 0.13, /var is 2.02, /run/user/1001 is 0.00, /dev/shm is 0.00, /run is 0.05, /run/lock is 0.00, /space is 0.24, /run/user/1003 is 0.00, /tmp is 0.02, / is 8.07, /space2 is 7.44, /mnt/usb1 is 0.51, /export/oxa is 0.01, /dev is 0.03, /sys/fs/cgroup is 0.00.

08h50

infra :: backup2.infra :: Inode usage in percent OKs: /run/user/1001 is 0.00, /tmp is 0.02, /dev is 0.03, /run/user/0 is 0.00, /dev/shm is 0.00, /run is 0.05, /space is 0.24, /sys/fs/cgroup is 0.00, /mnt/usb1 is 0.51, / is 8.07, /home is 0.13, /space2 is 7.44, /run/lock is 0.00, /var is 2.02, /export/oxa is 0.01.

infra :: backup2.infra :: Disk usage in percent OKs: / is 18.46, /mnt/usb1 is 30.62, /sys/fs/cgroup is 0.00, /export/oxa is 51.62, /run/lock is 0.00, /var is 22.29, /space2 is 40.39, /home is 8.87, /tmp is 0.60, /run/user/1001 is 0.00, /space is 76.49, /dev/shm is 0.00, /run is 9.27, /run/user/0 is 0.00.

All is Ok, no error in master logs, but still I received lots of these messages.

Here are the logs on master about this node

munin-update.log:2016/03/25 10:40:24 [WARNING] Service nfs4_client on backup2.infra/backup2.admin2:4949 returned no data for label fsinfo munin-update.log:2016/03/25 10:40:21 [WARNING] Service nfs_client on backup2.infra/backup2.admin2:4949 returned no data for label remove

munin-update.log:2016/03/25 09:55:06 [INFO] starting work in 29082 for backup2.infra/backup2.admin2:4949. munin-update.log:2016/03/25 09:55:06 [INFO] node backup2.infra advertised itself as backup2 instead. munin-update.log:2016/03/25 09:55:12 [INFO]: Munin-update finished for node infra;backup2.infra (6.67 sec) munin-update.log:2016/03/25 09:55:13 [INFO] Reaping Munin::Master::UpdateWorker. Exit value/signal: 0/0

Configuration for notification

 contact.devs.command mail -s "Munin notification ${var:host}" my@mail.com 
 contact.devs.always_send warning critical

Here is the configuration file for this node (generated, as for all nodes)

[backup2.infra]
     address backup2.admin2
     use_node_name yes
     diskstats_latency.backup2_store_export.avgrdwait.warning :7
     diskstats_latency.backup2_store_export.avgwrwait.warning :7
     diskstats_latency.backup2_store_export.avgrdwait.critical :10
     diskstats_latency.backup2_store_export.avgwrwait.critical :10

Munin Master and node version: 2.0.25-1 (both Debian Jessie)

Where can I watch to undestand and resolve ?

6

The df plugin in Debian also checks the dynamically mounted filesystems under /run/user/<uid> which appear when a user logs in and which disappear when the user logs out. Even though all levels are OK, this appearance and disappearance is considered a change which triggers an email.

You should be able to avoid this by creating a file called /etc/munin/plugin-conf.d/df with the following content:

[df*]
env.exclude_re /run/user/

To check if your settings work and to list which paths the df plugin considers, use the following command:

munin-run -d df

If you are happy with the outcome, restart the munin-node service (service munin-node restart).

  • How is this done for a specific host only and not for all users of df? – Zitrax Apr 10 at 11:28
1

Recent Munin in Debian and derived distributions should handle this according to Debian bug #788736.

Some logic around tmpfs type mounts (which /run/user/* is) has been fixed in the Munin upstream project. As far as I can see, they are however not excluded pr. default (probably the Debian specific configuration that does this).

0

For me, docker volumes were another cause of this error.

I've ended up using this configuration to fix both the @Oliver's issue with /run/user as well as my docker one;

[df*]
env.exclude_re ^(/run/user/|/var/lib/docker)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.