Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I want to subscribe a daemon, inotify-style, to receive a notification when free space on a given filesystem dips below a certain percentage. Is this possible?

share|improve this question
    
There exists no interface in default kernels to generate events that do this. The best you can do is poll a file frequently. –  Matthew Ife May 7 '12 at 18:20
    
There actually might be a novel way of doing this event based.. will update with an answer if I can do a proof of concept. –  Matthew Ife May 7 '12 at 19:09
    
Thanks for checking it out! –  weaver May 7 '12 at 19:51
    
that will do what you want but you'll need to add some extra effort from inotify of your own to get it doing <whatever you want> on the event coming true. –  Matthew Ife May 7 '12 at 22:14
    
This is not an answer, but a comment to answer from MIfe (that idiotic "reputation" thing does not allow me to add comments). The following line appears to be incorrect: fanotify_init(FAN_CLASS_NOTIF, FAN_CLOEXEC) See manpage: spinics.net/lists/linux-man/msg02302.html also see samples at: lanedo.com/~aleksander/fanotify which has the following line: fanotify_init(FAN_CLOEXEC, O_RDONLY | O_CLOEXEC | O_LARGEFILE) –  jhnlmn Jun 3 '12 at 3:10
add comment

3 Answers

up vote 1 down vote accepted

This works as an event based mechanism. I've not ran it for long periods so wont guarantee its stability.

This uses a very recent system call API called fanotify. Probably need a 2.6.37 kernel or greater to run it (so EL5 is out of the question, for example). If you get complaints it wont compile, its probably too old a kernel.

It compiles with:

gcc -o notifier notifier.c

The way it works is as such:-

./notifier /home/file /dev/shm/monit 10

Arguments are as follows:

  1. A file on the filesystem you want to monitor.
  2. A path to a file that will be created if you go over threshold (and be deleted if under)
  3. A percentage of free space that should be available to be under threshold.

This will setup the monitor. Every closed file handle on the filesystem that had a write flag open initiates the event check.

#include <stdio.h>
#include <stdint.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <errno.h>
#include <limits.h>
#include <linux/fanotify.h>
#include <sys/stat.h>
#include <sys/statvfs.h>
#include <fcntl.h>

int main(const int argc, const char **argv) {
    if (argc < 4) {
        fprintf(stderr, "Supply a path to a file on the mountpoint to listen to, a monitor file and a free %% threshold..\n");
        exit(1);
    }

    if (access(argv[1], R_OK) < 0) {
        fprintf(stderr, "Unable to read file: %s\n", strerror(errno));
        exit(1);
    }

    int len, rc;
    unsigned char donestat = 0,  alerted = 0;
        const char *path = argv[1];
    const char *monpath = argv[2];
    int threshold = atoi(argv[3]);
    char buf[4096];
    struct fanotify_event_metadata *fem = NULL;
    int fan_fd = -1;
    uint64_t mask = FAN_CLOSE_WRITE;
    struct statvfs sfs;
    float bfree;

    memset(&sfs, 0, sizeof(sfs));
    unlink(monpath);

    if (threshold <= 0 || threshold >= 100) {
        fprintf(stderr, "Incorrect threshold provided");
        rc = 1;
        goto end;
    }

    fan_fd = fanotify_init(FAN_CLASS_NOTIF, FAN_CLOEXEC);
    if (fan_fd < 0) {
        perror("fanotify_init");
        rc = 1;
        goto end;
    }

    rc = fanotify_mark(fan_fd, FAN_MARK_ADD|FAN_MARK_MOUNT, mask, AT_FDCWD, path);
    if (rc < 0) {
        perror("fanotify_mark");
        rc = 1;
        goto end;
    }

    while ((len = read(fan_fd, buf, sizeof(buf))) > 0) {
        fem = (void *)buf;
        donestat = 0;

        while (FAN_EVENT_OK(fem, len)) {
            if (fem->vers < 2) {
                fprintf(stderr, "fanotify is too old\n");
                goto end;
            }

            if (!donestat) {
                rc = fstatvfs(fem->fd, &sfs);
                if (rc < 0) {
                    perror("fstatvfs");
                    rc = 1;
                    goto end;
                }
                bfree = 100 - (((float)(sfs.f_blocks - ((sfs.f_blocks - sfs.f_bfree))) / (float)(sfs.f_blocks)) * 100);
                if ((bfree < (float)threshold)) {
                    if (!alerted) {
                        creat(monpath, S_IRUSR|S_IWUSR);
                        alerted = 1;
                    }
                }
                else {
                    if (alerted) {
                        unlink(monpath);
                        alerted = 0;
                    }
                }
            }
            donestat = 1;
            close(fem->fd);
            fem = FAN_EVENT_NEXT(fem, len);
        }
    }
    if (len < 0) {
        perror("Read fan_fd");
        rc = 1;
        goto end;
    }

end:
    close(fan_fd);
    exit(rc);
}

From there you can use inotify to watch for the file to be created/deleted to know the outcome.

To test, set the threshold to something you know breaches it now and then touch a file on the affected filesystem. You should get your monitor file created.

Obviously its probably best to put the monitor file somewhere not on the same filesystem (/dev/shm is a good place).

share|improve this answer
    
Thanks very much for looking into this! I will try this and report back... –  weaver May 7 '12 at 22:29
add comment

You can use the Monit utility for this. You can set the interval with which this checks, but 60 seconds is the norm for polling.

The configuration for filesystem monitoring will look like:

check device root with path /
    if SPACE usage > 80% then alert

check device var with path /var
    if SPACE usage > 80% then alert
share|improve this answer
    
Thanks, but I'm looking for a solution that informs me about this in real time, so I can take action before the processes can encounter an out-of-space condition. Polling/recurrent solutions are insufficient. –  weaver May 7 '12 at 22:27
    
You can lower the polling interval. A better long term solution is something like OpenNMS, which has rising-threshold alerts that are triggered by a certain rate of increase versus a hard limit. –  ewwhite May 7 '12 at 22:57
add comment

Since there will be more things you want to monitor, you might want to spend some time learning nagios. It sounds like you had a case where you ran out of space, and you don't want that particular failure to happen again. But systems can fail in unexpected ways, and all services should be monitored.

In a pinch you can just use the plugins. They're easy to install, and do what they say (check_disk checks disk space). They return "not 0" on failure. Failure can be warning, critical or unknown.

apt-get install nagios-plugins

Add something like this to crontab will trigger $send_error on failure. It will be triggered if > 50% of partition "/" is used.

send_error="command you want to run on failure"
*/5 * * * *  /usr/lib/nagios/plugins/check_disk -w 50% -c 25% -p / || $send_error
share|improve this answer
    
Thanks, but I'm looking for a solution that informs me about this in real time, so I can take action before the processes can encounter an out-of-space condition. Polling/recurrent solutions are insufficient. –  weaver May 7 '12 at 22:27
    
So set a very short polling interval for the alerts that you are so careful about. I'm sure you can monitor disk space at 1 second intervals on a single machine without any performance issues. Since human beings can't react much quicker than that anyway, it makes the fastidious "difference" between "polling" and "notification" irrelevant. –  Domingo Ignacio Galdos Nov 25 '12 at 17:31
    
With that said, I'm very skeptical you really need to know within a second unless you have a pretty unusual workload. Even in professional situations, monitoring critical/revenue-impacting systems, I've never seen a monitoring interval lower than a minute. In real life, it just doesn't actually make enough of a difference to be worth the cost. With that said, as I mentioned, you're only monitoring 1 machine, not 10,000, so knock yourself out and set 1-second thresholds in Nagios! It's wasteful to rewrite everything yourself (unless you're doing so for learning/fun of course). –  Domingo Ignacio Galdos Nov 25 '12 at 17:35
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.