Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have an apache web server with various virtual hosts. Each virtual host logs to date stamped files using cronolog. The result is that I end up with a lot of files like this:

----rw-r-- 1 root     root      467894 2013-02-01 13:43 2013-02-01-tracg.dev.local-access.log
----rw-r-- 1 root     root         222 2013-02-01 01:09 2013-02-01-tracg.dev.local-error.log
----rw-r-- 1 root     root         222 2013-02-02 01:10 2013-02-02-tracg.dev.local-error.log
----rw-r-- 1 root     root         333 2013-02-03 06:47 2013-02-03-tracg.dev.local-error.log
-rw-r--r-- 1 root     root       39574 2013-02-04 14:20 2013-02-04-tracg.dev.local-access.log
----rw-r-- 1 root     root         444 2013-02-04 08:24 2013-02-04-tracg.dev.local-error.log
----rw-r-- 1 root     root         444 2013-02-05 12:14 2013-02-05-tracg.dev.local-error.log
----rw-r-- 1 root     root       98135 2013-02-06 13:30 2013-02-06-tracg.dev.local-access.log
-rw-r--r-- 1 root     root         666 2013-02-06 09:13 2013-02-06-tracg.dev.local-error.log
-rw-r--r-- 1 root     root        8872 2013-02-07 13:55 2013-02-07-tracg.dev.local-access.log
-rw-r--r-- 1 root     root         666 2013-02-07 02:00 2013-02-07-tracg.dev.local-error.log
-rw-r--r-- 1 root     root         666 2013-02-08 14:31 2013-02-08-tracg.dev.local-error.log
----rw-r-- 1 root     root         192 2013-02-09 01:08 2013-02-09-tracg.dev.local-access.log
-rw-r--r-- 1 root     root         530 2013-02-09 01:08 2013-02-09-tracg.dev.local-error.log
----rw-r-- 1 root     root         192 2013-02-10 01:08 2013-02-10-tracg.dev.local-access.log
-rw-r--r-- 1 root     root         641 2013-02-10 06:32 2013-02-10-tracg.dev.local-error.log
----rw-r-- 1 root     root       64565 2013-02-11 13:56 2013-02-11-tracg.dev.local-access.log
-rw-r--r-- 1 root     root         752 2013-02-11 06:28 2013-02-11-tracg.dev.local-error.log
----rw-r-- 1 root     root       54988 2013-02-12 11:56 2013-02-12-tracg.dev.local-access.log
-rw-r--r-- 1 root     root         752 2013-02-12 08:45 2013-02-12-tracg.dev.local-error.log
-rw-r--r-- 1 root     root       15003 2013-02-13 11:19 2013-02-13-tracg.dev.local-access.log
-rw-r--r-- 1 root     root         666 2013-02-13 10:48 2013-02-13-tracg.dev.local-error.log
----rw-r-- 1 root     root       32819 2013-02-14 15:43 2013-02-14-tracg.dev.local-access.log
-rw-r--r-- 1 root     root         666 2013-02-14 14:29 2013-02-14-tracg.dev.local-error.log
----rw-r-- 1 root     root      135665 2013-02-15 13:53 2013-02-15-tracg.dev.local-access.log
-rw-r--r-- 1 root     root         666 2013-02-15 11:39 2013-02-15-tracg.dev.local-error.log
-rw-r--r-- 1 root     root         444 2013-02-16 01:10 2013-02-16-tracg.dev.local-error.log
-rw-r--r-- 1 root     root         555 2013-02-17 06:33 2013-02-17-tracg.dev.local-error.log
-rw-r--r-- 1 root     root       30763 2013-02-18 14:15 2013-02-18-tracg.dev.local-access.log
-rw-r--r-- 1 root     root         666 2013-02-18 09:00 2013-02-18-tracg.dev.local-error.log
----rw-r-- 1 root     root       15058 2013-02-19 09:37 2013-02-19-tracg.dev.local-access.log
-rw-r--r-- 1 root     root         666 2013-02-19 09:16 2013-02-19-tracg.dev.local-error.log
----rw-r-- 1 root     root      132587 2013-02-20 15:55 2013-02-20-tracg.dev.local-access.log
-rw-r--r-- 1 root     root         444 2013-02-20 01:11 2013-02-20-tracg.dev.local-error.log
-rw-r--r-- 1 root     root         666 2013-02-21 09:07 2013-02-21-tracg.dev.local-error.log
-rw-r--r-- 1 root     root      226997 2013-02-22 15:36 2013-02-22-tracg.dev.local-access.log
-rw-r--r-- 1 root     root         666 2013-02-22 08:26 2013-02-22-tracg.dev.local-error.log
-rw-r--r-- 1 root     root         444 2013-02-23 01:12 2013-02-23-tracg.dev.local-error.log
-rw-r--r-- 1 root     root         555 2013-02-24 06:48 2013-02-24-tracg.dev.local-error.log
----rw-r-- 1 root     root      130913 2013-02-25 11:36 2013-02-25-tracg.dev.local-access.log
-rw-r--r-- 1 root     root         666 2013-02-25 11:37 2013-02-25-tracg.dev.local-error.log
----rw-r-- 1 root     root       21542 2013-02-26 14:43 2013-02-26-tracg.dev.local-access.log
-rw-r--r-- 1 root     root         666 2013-02-26 15:39 2013-02-26-tracg.dev.local-error.log
----rw-r-- 1 root     root       99698 2013-02-27 13:28 2013-02-27-tracg.dev.local-access.log
-rw-r--r-- 1 root     root         444 2013-02-27 01:13 2013-02-27-tracg.dev.local-error.log
----rw-r-- 1 root     root      111632 2013-02-28 13:57 2013-02-28-tracg.dev.local-access.log
-rw-r--r-- 1 root     root         666 2013-02-28 11:55 2013-02-28-tracg.dev.local-error.log
----rw-r-- 1 root     root      205731 2013-03-01 13:30 2013-03-01-tracg.dev.local-access.log
-rw-r--r-- 1 root     root        1332 2013-03-01 14:31 2013-03-01-tracg.dev.local-error.log
-rw-r--r-- 1 root     root         444 2013-03-02 01:11 2013-03-02-tracg.dev.local-error.log
-rw-r--r-- 1 root     root         555 2013-03-03 06:35 2013-03-03-tracg.dev.local-error.log
----rw-r-- 1 root     root      211616 2013-03-04 10:42 2013-03-04-tracg.dev.local-access.log
-rw-r--r-- 1 root     root         666 2013-03-04 10:42 2013-03-04-tracg.dev.local-error.log

This is all just fine, however, I would like to have a cron job automatically gzip all log files that are not currently in use. I cannot just do: gzip -9 /data/www/virtual//log/.log

This will also gzip the currently open log file. So how to do this, so that I gzip only the ones that are not currently in use?

share|improve this question
2  
This is why we use logrotate. –  Michael Hampton Mar 4 '13 at 10:00

4 Answers 4

up vote 2 down vote accepted

One way would be:

find /data/www/virtual/log/ -name '*.log' -mtime +2 -exec gzip {} +

This compresses any file that ends in .log that hasn't been modified for at least two days.

Another way would be:

gzip $(date --date="yesterday" "+/data/www/virtual/log/%Y-%m-*.log")

This calculates yesterday's date and compresses any files that start with that date and end in .log.

Bear in mind that $() and % won't work in cron jobs because $() is a bash thing and % has a special meaning in crontabs. Put this one in a script and call the script from the cron job. Actually, do that for all of these methods.

Yet another way:

for FILE in /data/www/virtual/log/*.log
do
  if [ `fuser -s $FILE` ]
  then
    gzip $FILE
  fi
done

This compresses any files that end in .log and are not currently open by any process.

It might be possible to configure logrotate to handle this for you while still using cronolog. I've never used logrotate with with log files that have a different name each day but you can specify *.log in the config files so it may be possible.

Alternatively, you could just ditch cronolog, log to the same file all the time and have logrotate move them around every day. This will also handle deleting the oldest ones so you have a relatively stable amount of disk space used.

share|improve this answer

Try:

ls -d /path/to/[0-9]*tracg.dev.local-*.log | sort -r | sed '1,2d' | xargs gzip -9

This would be faster than checking every file for being open, but it assumes that the first two listed (sorted reverse alphabetically; latest two will be at the top) are the ones you want to skip.

[ The sed command deletes lines 1-2 ]

share|improve this answer

Sample Bash script for moving old log files:

logdir='/data/www/virtual//log/'
year=`date +%Y`
touch -t `date +%m%d0000` /tmp/tdate
            for i in $(find $logdir -maxdepth 1 -type f ! -newer /tmp/tdate)
                    do
                            month=`ls -al $i | awk '{print $6}'`
                            day=`ls -al $i | awk '{print $7}'`
                            ls -al $i | awk '{print $8}' | grep ":" > /dev/null
                            if [ $? = 0 ]
                                    then
                                            mkdir -p $logdir/previous/$day-$month-$year
                                            mv $i $logdir/previous/$day-$month-$year
                                    else
                                            year1=`ls -al $i | awk '{print $8}'`
                                            mkdir -p $logdir/previous/$day-$month-$year1
                                            mv $i $logdir/previous/$day-$month-$year1
                            fi
                    done
    rm -rf /tmp/tdate

Create a directory previous in the /data/www/virtual//log/ directory. All the logs older than today's date will then be moved to the previous directory. Inside the previous directory it will again create directories and log files will reside in the directories of date of which they belonged.

Or a Simple solution, use Logrotate

share|improve this answer

logrotate accepts wildcards in the man file "man logrotate" there is a wildcard example for the INN usenet daemon:

   /var/log/news/* {
       monthly
       rotate 2
       olddir /var/log/news/old
       missingok
       postrotate
           kill -HUP ‘cat /var/run/inn.pid‘
       endscript
       nocompress
   }

Hope that helps

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.