log rotation tool that keeps only a specified amount of logs and discards everything else

Question

I'm looking for a cronolog-like tool that will keep only last n lines or last x minutes of logs piped to it and discard everything else

Is there such a beast?

UPDATE:

I know about logrotate and it renames and zips old logfiles, which is not what I want.

I want to discard old log lines and keep only recent lines.

Like i.e. doing this every so often: tail -10000 logfile > logfile.new mv logfile.new logfile except that with this technique you will most certainly lose log lines and you have to restart or otherwise signal the logging application to reopen the logfile.

Answer 1

Logrotate can be made to only keep one copy of a logfile... If you RTFM you'll find the following bit regarding configuration settings:

rotate count
    Log files are rotated count times before being removed or mailed to the 
    address specified in a mail directive. If count is  0,  old  versions  
    are  removed  rather than rotated.

You can couple rotate with size, again from the logrotate(8) man page, to keep the file size small. While not by number of lines but by k, M, G size.

size size
    Log  files  are rotated when they grow bigger than size bytes. If size is 
    followed by M, the size if assumed to be in megabytes.  If the G suffix is 
    used, the size  is  in gigabytes.   If  the k is used, the size is in 
    kilobytes. So size 100, size 100k, and size 100M are all valid.

Answer 2

You can use logrotate and put

tail -10000 logfile.0 > logfile.0.new 
mv logfile.0.new > logfile.0

as part of postrotate command. logrotate allows you to specify postrotate commands.

Answer 3

You will have to restart or signal the application anyway. The application somehow has to get to know the new offset for seek()ing or hast to reopen the filehandle when you trim the logfile.

Answer 4

Not entirely what you're saying, but you might check out logrotate(8). From the man page:

logrotate is designed to ease administration of systems that generate large numbers of log files. It allows automatic rotation, compression, removal, and mailing of log files. Each log file may be handled ... when it grows too large.

It's installed by default on RHEL and derivitives. I don't know about anything else like Ubuntu/derivitives or Windows.

Answer 5

At the end I've solved it like this (not the most elegant, but it works):

in apache (or whoever logs):

CustomLog "|/usr/local/cronolog/sbin/cronolog /var/tmp/mylog.%Y%m%d.log" logformat

in cron.daily:

find /var/tmp/ -name mylog* -mtime +$days | xargs --no-run-if-empty rm

this will delete old logs

and finally in the analyzing script:

lastdate=$(date -d "$INTERVAL sec ago" +%Y-%m-%dT%H:%M:%S)

grep -h $SEARCH /var/tmp/mylog* | awk -v lastdate="$lastdate" '$1>lastdate { print }' > /tmp/cutlog

and then work with /tmp/cutlog

the above example assumes ISO timestamps in the first field like: 2009-07-20T13:52:32

not the most elegant way but it does what I want. Maybe one day I'll write a feature for cronolog that would do the same thing :)