3

I want to subscribe a daemon, inotify-style, to receive a notification when free space on a given filesystem dips below a certain percentage. Is this possible?

4
  • There exists no interface in default kernels to generate events that do this. The best you can do is poll a file frequently. May 7, 2012 at 18:20
  • There actually might be a novel way of doing this event based.. will update with an answer if I can do a proof of concept. May 7, 2012 at 19:09
  • that will do what you want but you'll need to add some extra effort from inotify of your own to get it doing <whatever you want> on the event coming true. May 7, 2012 at 22:14
  • This is not an answer, but a comment to answer from MIfe (that idiotic "reputation" thing does not allow me to add comments). The following line appears to be incorrect: fanotify_init(FAN_CLASS_NOTIF, FAN_CLOEXEC) See manpage: spinics.net/lists/linux-man/msg02302.html also see samples at: lanedo.com/~aleksander/fanotify which has the following line: fanotify_init(FAN_CLOEXEC, O_RDONLY | O_CLOEXEC | O_LARGEFILE)
    – jhnlmn
    Jun 3, 2012 at 3:10

3 Answers 3

2

This works as an event based mechanism. I've not ran it for long periods so wont guarantee its stability.

This uses a very recent system call API called fanotify. Probably need a 2.6.37 kernel or greater to run it (so EL5 is out of the question, for example). If you get complaints it wont compile, its probably too old a kernel.

It compiles with:

gcc -o notifier notifier.c

The way it works is as such:-

./notifier /home/file /dev/shm/monit 10

Arguments are as follows:

  1. A file on the filesystem you want to monitor.
  2. A path to a file that will be created if you go over threshold (and be deleted if under)
  3. A percentage of free space that should be available to be under threshold.

This will setup the monitor. Every closed file handle on the filesystem that had a write flag open initiates the event check.

#include <stdio.h>
#include <stdint.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <errno.h>
#include <limits.h>
#include <linux/fanotify.h>
#include <sys/stat.h>
#include <sys/statvfs.h>
#include <fcntl.h>

int main(const int argc, const char **argv) {
    if (argc < 4) {
        fprintf(stderr, "Supply a path to a file on the mountpoint to listen to, a monitor file and a free %% threshold..\n");
        exit(1);
    }

    if (access(argv[1], R_OK) < 0) {
        fprintf(stderr, "Unable to read file: %s\n", strerror(errno));
        exit(1);
    }

    int len, rc;
    unsigned char donestat = 0,  alerted = 0;
        const char *path = argv[1];
    const char *monpath = argv[2];
    int threshold = atoi(argv[3]);
    char buf[4096];
    struct fanotify_event_metadata *fem = NULL;
    int fan_fd = -1;
    uint64_t mask = FAN_CLOSE_WRITE;
    struct statvfs sfs;
    float bfree;

    memset(&sfs, 0, sizeof(sfs));
    unlink(monpath);

    if (threshold <= 0 || threshold >= 100) {
        fprintf(stderr, "Incorrect threshold provided");
        rc = 1;
        goto end;
    }

    fan_fd = fanotify_init(FAN_CLASS_NOTIF, FAN_CLOEXEC);
    if (fan_fd < 0) {
        perror("fanotify_init");
        rc = 1;
        goto end;
    }

    rc = fanotify_mark(fan_fd, FAN_MARK_ADD|FAN_MARK_MOUNT, mask, AT_FDCWD, path);
    if (rc < 0) {
        perror("fanotify_mark");
        rc = 1;
        goto end;
    }

    while ((len = read(fan_fd, buf, sizeof(buf))) > 0) {
        fem = (void *)buf;
        donestat = 0;

        while (FAN_EVENT_OK(fem, len)) {
            if (fem->vers < 2) {
                fprintf(stderr, "fanotify is too old\n");
                goto end;
            }

            if (!donestat) {
                rc = fstatvfs(fem->fd, &sfs);
                if (rc < 0) {
                    perror("fstatvfs");
                    rc = 1;
                    goto end;
                }
                bfree = 100 - (((float)(sfs.f_blocks - ((sfs.f_blocks - sfs.f_bfree))) / (float)(sfs.f_blocks)) * 100);
                if ((bfree < (float)threshold)) {
                    if (!alerted) {
                        creat(monpath, S_IRUSR|S_IWUSR);
                        alerted = 1;
                    }
                }
                else {
                    if (alerted) {
                        unlink(monpath);
                        alerted = 0;
                    }
                }
            }
            donestat = 1;
            close(fem->fd);
            fem = FAN_EVENT_NEXT(fem, len);
        }
    }
    if (len < 0) {
        perror("Read fan_fd");
        rc = 1;
        goto end;
    }

end:
    close(fan_fd);
    exit(rc);
}

From there you can use inotify to watch for the file to be created/deleted to know the outcome.

To test, set the threshold to something you know breaches it now and then touch a file on the affected filesystem. You should get your monitor file created.

Obviously its probably best to put the monitor file somewhere not on the same filesystem (/dev/shm is a good place).

3
  • Thanks very much for looking into this! I will try this and report back...
    – weaver
    May 7, 2012 at 22:29
  • 1
    If this triggers on the file close event, it won't catch an event where a log file grows and fills the disk without ever being closed.
    – mc0e
    Mar 7, 2016 at 14:23
  • @matthew-ife should line 76 be bfree = (((float)(sfs.f_bfree) / (float)(sfs.f_blocks)) * 100); Because now the function is reversed. e.g. file is removed when space runs out and the limit is usage % not free %
    – Manwe
    Nov 22, 2016 at 11:00
4

You can use the Monit utility for this. You can set the interval with which this checks, but 60 seconds is the norm for polling.

The configuration for filesystem monitoring will look like:

check device root with path /
    if SPACE usage > 80% then alert

check device var with path /var
    if SPACE usage > 80% then alert
2
  • 1
    Thanks, but I'm looking for a solution that informs me about this in real time, so I can take action before the processes can encounter an out-of-space condition. Polling/recurrent solutions are insufficient.
    – weaver
    May 7, 2012 at 22:27
  • You can lower the polling interval. A better long term solution is something like OpenNMS, which has rising-threshold alerts that are triggered by a certain rate of increase versus a hard limit.
    – ewwhite
    May 7, 2012 at 22:57
1

Since there will be more things you want to monitor, you might want to spend some time learning nagios. It sounds like you had a case where you ran out of space, and you don't want that particular failure to happen again. But systems can fail in unexpected ways, and all services should be monitored.

In a pinch you can just use the plugins. They're easy to install, and do what they say (check_disk checks disk space). They return "not 0" on failure. Failure can be warning, critical or unknown.

apt-get install nagios-plugins

Add something like this to crontab will trigger $send_error on failure. It will be triggered if > 50% of partition "/" is used.

send_error="command you want to run on failure"
*/5 * * * *  /usr/lib/nagios/plugins/check_disk -w 50% -c 25% -p / || $send_error
3
  • Thanks, but I'm looking for a solution that informs me about this in real time, so I can take action before the processes can encounter an out-of-space condition. Polling/recurrent solutions are insufficient.
    – weaver
    May 7, 2012 at 22:27
  • So set a very short polling interval for the alerts that you are so careful about. I'm sure you can monitor disk space at 1 second intervals on a single machine without any performance issues. Since human beings can't react much quicker than that anyway, it makes the fastidious "difference" between "polling" and "notification" irrelevant. Nov 25, 2012 at 17:31
  • With that said, I'm very skeptical you really need to know within a second unless you have a pretty unusual workload. Even in professional situations, monitoring critical/revenue-impacting systems, I've never seen a monitoring interval lower than a minute. In real life, it just doesn't actually make enough of a difference to be worth the cost. With that said, as I mentioned, you're only monitoring 1 machine, not 10,000, so knock yourself out and set 1-second thresholds in Nagios! It's wasteful to rewrite everything yourself (unless you're doing so for learning/fun of course). Nov 25, 2012 at 17:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.