Free-space driven log rotation on linux?

Question

Someone just asked me 'how long should we keep logs for our application', and my answer was 'until the disk is full' as there's no reason to throw them away other than running out of space.

However, standard logrotate wants us to specify a specific period + number of rotations. Is there something similar that would let us say "rotate daily, and keep as much history as you like until there is only 5% space free"?

The platform is Redhat Linux.

Answer 1

You can perhaps use firstaction or lastaction directives to call a shell script that tests for disk free space, and then run a delete on the oldest files.

   firstaction/endscript
          The lines between firstaction and endscript (both of which must appear on lines by themselves) are
          executed (using /bin/sh) once before all log files that match the wildcarded pattern are  rotated,
          before  prerotate  script  is  run  and  only if at least one log will actually be rotated.  These
          directives may only appear inside a log file definition. Whole pattern is passed to the script  as
          first  argument.  If  the script exits with error, no further processing is done. See also lastac-
          tion.

Update:

Here's a Stackoverflow post on the type of script you can run:

https://stackoverflow.com/questions/7523059/remove-oldest-file-in-repository

Answer 2

logrotate itself has no such option. You can add a cron script that find the oldest log to remove whenever the free space falls below your criteria. You can do some other validation as well. However, getting disk too full all the time is not a good idea because the system will not be able to create large temporary files and could cause application failures.

Answer 3

I just wanted to point out there are cases where you don't want your logs to fill all available disk space. I have dealt with several hosts with thin provisioned /var directories and keeping the logs to a certain size was crucial. We used a cronies job in conjunction with logrorate to keep the size down. Something similar could be used in your environment, although a central log server like splunk or syslog-ng would probably be a better option.

Answer 4

As @cjc suggested, you can use firstaction. See this example:

/mnt/user/logs/*.log  /mnt/user/logs/*/*.log {
        daily
        missingok
        rotate 7
        compress
        delaycompress
        notifempty
        su root www-data
        create 760 root www-data
        firstaction
          for file in `find  -type f -size +1024M`; do
              percent=`df -h | grep /mnt/user | awk '{print $5}' | sed 's/%//'`
              if [ $percent -gt 50 ]; then 
                  echo "Removed $file" >> /mnt/user/logs/logrotate.log
                  rm $file
              fi
           done;
        endscript
}

In this example you removed the files bigger than 1GB from the /mnt/user partition if the partition used space is more than the 50%.