How to compress and clean logs with logrotate but not rotate them

Question

I have a Tomcat server which create access logs via a valve (org.apache.catalina.valves.FastCommonAccessLogValve). This valve takes care of rotating the access log file, but not of compressing them or deleting them after some time.

For the moment, I have a cron job that uses find [...] -mtime +30 [...] to compress and delete the logs. I would rather use logrotate, so that log rotation is in a centralized place for all logs. I dont like having a seperate solution just for Tomcat.

I tried to read logrotate documentation, but I am still a bit lost. Can I use logrotate just to compress and cleanup log files ? How would I do that ?

Or turning the problem around, is there a Tomcat access log valve that will compress and cleanup log files ?

Thank for your help !

Answer 1

Fairly simple one that i have seen works like this.

Create a file in /etc/logrotate.d called tomcat containing the following:-

/var/log/tomcat/catalina.out { 
  copytruncate 
  daily 
  rotate 7 
  compress 
  missingok 
}

This runs daily, compresses the file, and keeps 7 days worth (rotate 7). copytruncate means that it'll copy then truncate the original file so that there is no need to restart tomcat. missing ok will not error if it's not there.

The access.log Valve can be changed to not rotate by adding rotatable=false:-

<Valve className="org.apache.catalina.valves.AccessLogValve
     ...
     ...
     suffix="log" rotatable="false" />

Answer 2

Modified TimP's script--Added deleting very old files, added scan for old compressed files.

#!/bin/bash
#
# TPP 2013-02-21
# RJK 2014-08-14
#
# Tomcat apps use a variety of loggers, mostly log4j.
# These rotate, conflicting with logrotate, the unix log rotation system.
#
# Some files eg catalina.out
# are rotated to a backup containing a date eg catalina.2013-01-06.log
# which can then be compressed with bz2 to catalina.2013-01-06.log.bz2
# or removed if older than a given number of days(MTIME).
#
cd /var/log/tomcat6
# 2013-02-21
DATE=`date --rfc-3339=date`
YEAR=`date +%Y`
MILLENIUM=20
# 2014-08-14
MTIME=14
# 2014-08-14
#for f in $(find catalina* |grep -v bz2 |grep -v '$DATE' |grep $YEAR)
for f in $(find catalina* |grep -v bz2 |grep -v '$DATE' |grep $MILLENIUM)
do
 # 2014-08-14
 if test `find $f -mtime +$MTIME`
 then
   echo "rm -f $f"
   rm -f $f
 else
   echo "bzip2 $f"
   bzip2 $f
 fi
done
# However others are active whilst containing a date
# so we will find all and not compress the most recent
for l in 'localhost*' 'opt-db*' 'opt*' 'host-manager*' 'manager*'
do
 export previous=
 for f in $(find $l |grep -v bz2 |sort)
 do
  if [ "${previous}" != "" ]
  then
    echo "bzip2 ${previous}"
    bzip2 $previous
  fi
  export previous=$f
 done
done
# 2014-08-14
for f in $(find *bz2)
do
 if test `find $f -mtime +$MTIME`
 then
   echo "rm -f $f"
   rm -f $f
 fi
done
exit 0 

Answer 3

It's surprisingly simple. Just tell logrotate which files you specifically want to rotate. nocreate tells logrotate not to recreate an empty file after moving the old one (if you're rotating files into a subfolder).

/var/log/tomcat/catalina.out.* { 
  daily 
  nocreate
  compress 
  missingok 
}

Answer 4

I did not wish to change the Tomcat configuration, so created a script which compressed the rotated files

#!/bin/bash
#
# TPP 2013-02-21
#
# Tomcat apps use a variety of loggers, mostly log4j.
# These rotate, conflicting with logrotate, the unix log rotation system. 
#
# Some files eg catalina.out
# are rotated to a backup containing a date eg catalina.2013-01-06.log
# which can then be compressed with bz2 to catalina.2013-01-06.log.bz2
#

cd /var/log/tomcat6

# 2013-02-21
DATE=`date --rfc-3339=date`
YEAR=`date +%Y`

for f in $(find catalina* |grep -v bz2 |grep -v '$DATE' |grep $YEAR)
do
 echo "bzip2 $f" 
 bzip2 $f
done

# However others are active whilst containing a date
# so we will find all and not compress the most recent
for l in 'localhost*' 'opt-db*' 'opt*' 'host-manager*' 'manager*'
do
 export previous=
 for f in $(find $l |grep -v bz2 |sort)
 do
  if [ "${previous}" != "" ]
  then
    echo "bzip2 ${previous}" 
    bzip2 $previous
  fi
  export previous=$f
 done
done

exit 0

Answer 5

For local_access_log.YYYY-MM-DD.txt compression I wrote this script after seeing this post :-

#!/bin/bash
#
# If Tomcat uses server.xml config to rotate localhost_access_log,
# the daily rotated logs will need compressing and old ones deleted to stop filling
# the log partiton. Cannot use the system logrotate command as conficts with tomcat rotate
# therefore run this script in a daily cronjob
#
# localhost_access_log.2015-09-03.txt
#
# Add this script in /etc/cron.daily/ owned by root
#

CATALINA_BASE=`ps aux | grep catalina.base | awk -F'catalina.base\=' '{print $2}' | awk '{print $1}'`

if [ ! $CATALINA_BASE ]
then
    if [ -r /var/lib/tomcat8 ]
    then
        CATALINA_BASE=/var/lib/tomcat8
    else
        echo "Error: cannot find CATALINA_BASE"
        exit 1
    fi
fi

cd ${CATALINA_BASE}/logs

if [ $? -ne 0 ]
then
    echo "Error, cannot cd to logs directory, quitting...."
    exit 1
fi

# today's date (not to be gzipped)
DATE=`date --rfc-3339=date`
# number of days to keep
MTIME=28

# Compress all previous days uncompressed logs
for log in `ls localhost_access_log* | grep -v bz2 | grep -v $DATE`
do
    bzip2 $log
done

# delete old logs
find . -name "*.bz2" -mtime +$MTIME -exec rm {} \;

exit 0

Answer 6

A simple solution provided here. This also includes the s3 upload option.

https://yottabrain.org/tomcat/tomcat-log-backup/