Apache2 logs are not rotating

Question

My apache log files on my Ubuntu 14.04 machine in the /var/log/apache2/ do not seem to be rotating. I had this problem in the past and had to manually rotate the log file, but it looks like it still is not rotating automatically.

I ran sudo logrotate -d /etc/logrotate.conf to try and manually rotate the log in case the cron was having issues. I also ran this as root to avoid permission issues if this was the case. Still did not rotate, despite the output appearing like it did. Here is the relevant information logrotate returned:

rotating pattern: /var/log/apache2/*.log  weekly (52 rotations)
empty log files are not rotated, old logs are removed
switching euid to 0 and egid to 104
considering log /var/log/apache2/access.log
  log needs rotating
considering log /var/log/apache2/error.log
  log needs rotating
rotating log /var/log/apache2/access.log, log->rotateCount is 52
dateext suffix '-20160404'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
compressing log with: /bin/gzip
renaming /var/log/apache2/access.log.52.gz to /var/log/apache2/access.log.53.gz (rotatecount 52, logstart 1, i 52),
renaming /var/log/apache2/access.log.51.gz to /var/log/apache2/access.log.52.gz (rotatecount 52, logstart 1, i 51),
renaming /var/log/apache2/access.log.50.gz to /var/log/apache2/access.log.51.gz (rotatecount 52, logstart 1, i 50),
renaming /var/log/apache2/access.log.49.gz to /var/log/apache2/access.log.50.gz (rotatecount 52, logstart 1, i 49),
renaming /var/log/apache2/access.log.48.gz to /var/log/apache2/access.log.49.gz (rotatecount 52, logstart 1, i 48),
renaming /var/log/apache2/access.log.47.gz to /var/log/apache2/access.log.48.gz (rotatecount 52, logstart 1, i 47),
renaming /var/log/apache2/access.log.46.gz to /var/log/apache2/access.log.47.gz (rotatecount 52, logstart 1, i 46),
renaming /var/log/apache2/access.log.45.gz to /var/log/apache2/access.log.46.gz (rotatecount 52, logstart 1, i 45),
renaming /var/log/apache2/access.log.44.gz to /var/log/apache2/access.log.45.gz (rotatecount 52, logstart 1, i 44),
renaming /var/log/apache2/access.log.43.gz to /var/log/apache2/access.log.44.gz (rotatecount 52, logstart 1, i 43),
renaming /var/log/apache2/access.log.42.gz to /var/log/apache2/access.log.43.gz (rotatecount 52, logstart 1, i 42),
renaming /var/log/apache2/access.log.41.gz to /var/log/apache2/access.log.42.gz (rotatecount 52, logstart 1, i 41),
renaming /var/log/apache2/access.log.40.gz to /var/log/apache2/access.log.41.gz (rotatecount 52, logstart 1, i 40),
renaming /var/log/apache2/access.log.39.gz to /var/log/apache2/access.log.40.gz (rotatecount 52, logstart 1, i 39),
renaming /var/log/apache2/access.log.38.gz to /var/log/apache2/access.log.39.gz (rotatecount 52, logstart 1, i 38),
renaming /var/log/apache2/access.log.37.gz to /var/log/apache2/access.log.38.gz (rotatecount 52, logstart 1, i 37),
renaming /var/log/apache2/access.log.36.gz to /var/log/apache2/access.log.37.gz (rotatecount 52, logstart 1, i 36),
renaming /var/log/apache2/access.log.35.gz to /var/log/apache2/access.log.36.gz (rotatecount 52, logstart 1, i 35),
renaming /var/log/apache2/access.log.34.gz to /var/log/apache2/access.log.35.gz (rotatecount 52, logstart 1, i 34),
renaming /var/log/apache2/access.log.33.gz to /var/log/apache2/access.log.34.gz (rotatecount 52, logstart 1, i 33),
renaming /var/log/apache2/access.log.32.gz to /var/log/apache2/access.log.33.gz (rotatecount 52, logstart 1, i 32),
renaming /var/log/apache2/access.log.31.gz to /var/log/apache2/access.log.32.gz (rotatecount 52, logstart 1, i 31),
renaming /var/log/apache2/access.log.30.gz to /var/log/apache2/access.log.31.gz (rotatecount 52, logstart 1, i 30),
renaming /var/log/apache2/access.log.29.gz to /var/log/apache2/access.log.30.gz (rotatecount 52, logstart 1, i 29),
renaming /var/log/apache2/access.log.28.gz to /var/log/apache2/access.log.29.gz (rotatecount 52, logstart 1, i 28),
renaming /var/log/apache2/access.log.27.gz to /var/log/apache2/access.log.28.gz (rotatecount 52, logstart 1, i 27),
renaming /var/log/apache2/access.log.26.gz to /var/log/apache2/access.log.27.gz (rotatecount 52, logstart 1, i 26),
renaming /var/log/apache2/access.log.25.gz to /var/log/apache2/access.log.26.gz (rotatecount 52, logstart 1, i 25),
renaming /var/log/apache2/access.log.24.gz to /var/log/apache2/access.log.25.gz (rotatecount 52, logstart 1, i 24),
renaming /var/log/apache2/access.log.23.gz to /var/log/apache2/access.log.24.gz (rotatecount 52, logstart 1, i 23),
renaming /var/log/apache2/access.log.22.gz to /var/log/apache2/access.log.23.gz (rotatecount 52, logstart 1, i 22),
renaming /var/log/apache2/access.log.21.gz to /var/log/apache2/access.log.22.gz (rotatecount 52, logstart 1, i 21),
renaming /var/log/apache2/access.log.20.gz to /var/log/apache2/access.log.21.gz (rotatecount 52, logstart 1, i 20),
renaming /var/log/apache2/access.log.19.gz to /var/log/apache2/access.log.20.gz (rotatecount 52, logstart 1, i 19),
renaming /var/log/apache2/access.log.18.gz to /var/log/apache2/access.log.19.gz (rotatecount 52, logstart 1, i 18),
renaming /var/log/apache2/access.log.17.gz to /var/log/apache2/access.log.18.gz (rotatecount 52, logstart 1, i 17),
renaming /var/log/apache2/access.log.16.gz to /var/log/apache2/access.log.17.gz (rotatecount 52, logstart 1, i 16),
renaming /var/log/apache2/access.log.15.gz to /var/log/apache2/access.log.16.gz (rotatecount 52, logstart 1, i 15),
renaming /var/log/apache2/access.log.14.gz to /var/log/apache2/access.log.15.gz (rotatecount 52, logstart 1, i 14),
renaming /var/log/apache2/access.log.13.gz to /var/log/apache2/access.log.14.gz (rotatecount 52, logstart 1, i 13),
renaming /var/log/apache2/access.log.12.gz to /var/log/apache2/access.log.13.gz (rotatecount 52, logstart 1, i 12),
renaming /var/log/apache2/access.log.11.gz to /var/log/apache2/access.log.12.gz (rotatecount 52, logstart 1, i 11),
renaming /var/log/apache2/access.log.10.gz to /var/log/apache2/access.log.11.gz (rotatecount 52, logstart 1, i 10),
renaming /var/log/apache2/access.log.9.gz to /var/log/apache2/access.log.10.gz (rotatecount 52, logstart 1, i 9),
renaming /var/log/apache2/access.log.8.gz to /var/log/apache2/access.log.9.gz (rotatecount 52, logstart 1, i 8),
renaming /var/log/apache2/access.log.7.gz to /var/log/apache2/access.log.8.gz (rotatecount 52, logstart 1, i 7),
renaming /var/log/apache2/access.log.6.gz to /var/log/apache2/access.log.7.gz (rotatecount 52, logstart 1, i 6),
renaming /var/log/apache2/access.log.5.gz to /var/log/apache2/access.log.6.gz (rotatecount 52, logstart 1, i 5),
renaming /var/log/apache2/access.log.4.gz to /var/log/apache2/access.log.5.gz (rotatecount 52, logstart 1, i 4),
renaming /var/log/apache2/access.log.3.gz to /var/log/apache2/access.log.4.gz (rotatecount 52, logstart 1, i 3),
renaming /var/log/apache2/access.log.2.gz to /var/log/apache2/access.log.3.gz (rotatecount 52, logstart 1, i 2),
renaming /var/log/apache2/access.log.1.gz to /var/log/apache2/access.log.2.gz (rotatecount 52, logstart 1, i 1),
renaming /var/log/apache2/access.log.0.gz to /var/log/apache2/access.log.1.gz (rotatecount 52, logstart 1, i 0),
rotating log /var/log/apache2/error.log, log->rotateCount is 52
dateext suffix '-20160404'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
compressing log with: /bin/gzip
renaming /var/log/apache2/error.log.52.gz to /var/log/apache2/error.log.53.gz (rotatecount 52, logstart 1, i 52),
renaming /var/log/apache2/error.log.51.gz to /var/log/apache2/error.log.52.gz (rotatecount 52, logstart 1, i 51),
renaming /var/log/apache2/error.log.50.gz to /var/log/apache2/error.log.51.gz (rotatecount 52, logstart 1, i 50),
renaming /var/log/apache2/error.log.49.gz to /var/log/apache2/error.log.50.gz (rotatecount 52, logstart 1, i 49),
renaming /var/log/apache2/error.log.48.gz to /var/log/apache2/error.log.49.gz (rotatecount 52, logstart 1, i 48),
renaming /var/log/apache2/error.log.47.gz to /var/log/apache2/error.log.48.gz (rotatecount 52, logstart 1, i 47),
renaming /var/log/apache2/error.log.46.gz to /var/log/apache2/error.log.47.gz (rotatecount 52, logstart 1, i 46),
renaming /var/log/apache2/error.log.45.gz to /var/log/apache2/error.log.46.gz (rotatecount 52, logstart 1, i 45),
renaming /var/log/apache2/error.log.44.gz to /var/log/apache2/error.log.45.gz (rotatecount 52, logstart 1, i 44),
renaming /var/log/apache2/error.log.43.gz to /var/log/apache2/error.log.44.gz (rotatecount 52, logstart 1, i 43),
renaming /var/log/apache2/error.log.42.gz to /var/log/apache2/error.log.43.gz (rotatecount 52, logstart 1, i 42),
renaming /var/log/apache2/error.log.41.gz to /var/log/apache2/error.log.42.gz (rotatecount 52, logstart 1, i 41),
renaming /var/log/apache2/error.log.40.gz to /var/log/apache2/error.log.41.gz (rotatecount 52, logstart 1, i 40),
renaming /var/log/apache2/error.log.39.gz to /var/log/apache2/error.log.40.gz (rotatecount 52, logstart 1, i 39),
renaming /var/log/apache2/error.log.38.gz to /var/log/apache2/error.log.39.gz (rotatecount 52, logstart 1, i 38),
renaming /var/log/apache2/error.log.37.gz to /var/log/apache2/error.log.38.gz (rotatecount 52, logstart 1, i 37),
renaming /var/log/apache2/error.log.36.gz to /var/log/apache2/error.log.37.gz (rotatecount 52, logstart 1, i 36),
renaming /var/log/apache2/error.log.35.gz to /var/log/apache2/error.log.36.gz (rotatecount 52, logstart 1, i 35),
renaming /var/log/apache2/error.log.34.gz to /var/log/apache2/error.log.35.gz (rotatecount 52, logstart 1, i 34),
renaming /var/log/apache2/error.log.33.gz to /var/log/apache2/error.log.34.gz (rotatecount 52, logstart 1, i 33),
renaming /var/log/apache2/error.log.32.gz to /var/log/apache2/error.log.33.gz (rotatecount 52, logstart 1, i 32),
renaming /var/log/apache2/error.log.31.gz to /var/log/apache2/error.log.32.gz (rotatecount 52, logstart 1, i 31),
renaming /var/log/apache2/error.log.30.gz to /var/log/apache2/error.log.31.gz (rotatecount 52, logstart 1, i 30),
renaming /var/log/apache2/error.log.29.gz to /var/log/apache2/error.log.30.gz (rotatecount 52, logstart 1, i 29),
renaming /var/log/apache2/error.log.28.gz to /var/log/apache2/error.log.29.gz (rotatecount 52, logstart 1, i 28),
renaming /var/log/apache2/error.log.27.gz to /var/log/apache2/error.log.28.gz (rotatecount 52, logstart 1, i 27),
renaming /var/log/apache2/error.log.26.gz to /var/log/apache2/error.log.27.gz (rotatecount 52, logstart 1, i 26),
renaming /var/log/apache2/error.log.25.gz to /var/log/apache2/error.log.26.gz (rotatecount 52, logstart 1, i 25),
renaming /var/log/apache2/error.log.24.gz to /var/log/apache2/error.log.25.gz (rotatecount 52, logstart 1, i 24),
renaming /var/log/apache2/error.log.23.gz to /var/log/apache2/error.log.24.gz (rotatecount 52, logstart 1, i 23),
renaming /var/log/apache2/error.log.22.gz to /var/log/apache2/error.log.23.gz (rotatecount 52, logstart 1, i 22),
renaming /var/log/apache2/error.log.21.gz to /var/log/apache2/error.log.22.gz (rotatecount 52, logstart 1, i 21),
renaming /var/log/apache2/error.log.20.gz to /var/log/apache2/error.log.21.gz (rotatecount 52, logstart 1, i 20),
renaming /var/log/apache2/error.log.19.gz to /var/log/apache2/error.log.20.gz (rotatecount 52, logstart 1, i 19),
renaming /var/log/apache2/error.log.18.gz to /var/log/apache2/error.log.19.gz (rotatecount 52, logstart 1, i 18),
renaming /var/log/apache2/error.log.17.gz to /var/log/apache2/error.log.18.gz (rotatecount 52, logstart 1, i 17),
renaming /var/log/apache2/error.log.16.gz to /var/log/apache2/error.log.17.gz (rotatecount 52, logstart 1, i 16),
renaming /var/log/apache2/error.log.15.gz to /var/log/apache2/error.log.16.gz (rotatecount 52, logstart 1, i 15),
renaming /var/log/apache2/error.log.14.gz to /var/log/apache2/error.log.15.gz (rotatecount 52, logstart 1, i 14),
renaming /var/log/apache2/error.log.13.gz to /var/log/apache2/error.log.14.gz (rotatecount 52, logstart 1, i 13),
renaming /var/log/apache2/error.log.12.gz to /var/log/apache2/error.log.13.gz (rotatecount 52, logstart 1, i 12),
renaming /var/log/apache2/error.log.11.gz to /var/log/apache2/error.log.12.gz (rotatecount 52, logstart 1, i 11),
renaming /var/log/apache2/error.log.10.gz to /var/log/apache2/error.log.11.gz (rotatecount 52, logstart 1, i 10),
renaming /var/log/apache2/error.log.9.gz to /var/log/apache2/error.log.10.gz (rotatecount 52, logstart 1, i 9),
renaming /var/log/apache2/error.log.8.gz to /var/log/apache2/error.log.9.gz (rotatecount 52, logstart 1, i 8),
renaming /var/log/apache2/error.log.7.gz to /var/log/apache2/error.log.8.gz (rotatecount 52, logstart 1, i 7),
renaming /var/log/apache2/error.log.6.gz to /var/log/apache2/error.log.7.gz (rotatecount 52, logstart 1, i 6),
renaming /var/log/apache2/error.log.5.gz to /var/log/apache2/error.log.6.gz (rotatecount 52, logstart 1, i 5),
renaming /var/log/apache2/error.log.4.gz to /var/log/apache2/error.log.5.gz (rotatecount 52, logstart 1, i 4),
renaming /var/log/apache2/error.log.3.gz to /var/log/apache2/error.log.4.gz (rotatecount 52, logstart 1, i 3),
renaming /var/log/apache2/error.log.2.gz to /var/log/apache2/error.log.3.gz (rotatecount 52, logstart 1, i 2),
renaming /var/log/apache2/error.log.1.gz to /var/log/apache2/error.log.2.gz (rotatecount 52, logstart 1, i 1),
renaming /var/log/apache2/error.log.0.gz to /var/log/apache2/error.log.1.gz (rotatecount 52, logstart 1, i 0),
rotating log /var/log/apache2/other_vhosts_access.log, log->rotateCount is 52
running prerotate script
running script with arg /var/log/apache2/*.log : "
                if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
                        run-parts /etc/logrotate.d/httpd-prerotate; \
                fi; \
"
renaming /var/log/apache2/access.log to /var/log/apache2/access.log.1
creating new /var/log/apache2/access.log mode = 0640 uid = 0 gid = 4
renaming /var/log/apache2/error.log to /var/log/apache2/error.log.1
creating new /var/log/apache2/error.log mode = 0640 uid = 0 gid = 4
renaming /var/log/apache2/other_vhosts_access.log to /var/log/apache2/other_vhosts_access.log.1
creating new /var/log/apache2/other_vhosts_access.log mode = 0640 uid = 0 gid = 4
running postrotate script
running script with arg /var/log/apache2/*.log : "
                if /etc/init.d/apache2 status > /dev/null ; then \
                    /etc/init.d/apache2 reload > /dev/null; \
                fi;
"
removing old log /var/log/apache2/access.log.53.gz
error: error opening /var/log/apache2/access.log.53.gz: No such file or directory
switching euid to 0 and egid to 0

Both the error.log and access.log share the same problem. If you look at the access log files you'll notice the only one ever being touched is the access.log file and nothing is being rotated:

-rw-r----- 1 root adm  1.1G Apr  5 00:10 access.log
-rw-r----- 1 root adm  165M Dec  6 00:12 access.log.1
-rw-r----- 1 root adm   18M Oct  5  2015 access.log.10.gz
-rw-r----- 1 root adm   14M Sep 27  2015 access.log.11.gz
-rw-r----- 1 root adm   19M Sep 21  2015 access.log.12.gz
-rw-r----- 1 root adm   16M Sep 13  2015 access.log.13.gz
-rw-r----- 1 root adm   14M Sep  6  2015 access.log.14.gz
-rw-r----- 1 root adm   19M Aug 31  2015 access.log.15.gz
-rw-r----- 1 root adm   15M Aug 23  2015 access.log.16.gz
-rw-r----- 1 root adm   20M Aug 17  2015 access.log.17.gz
-rw-r----- 1 root adm   15M Aug  9  2015 access.log.18.gz
-rw-r----- 1 root adm   20M Aug  3  2015 access.log.19.gz
-rw-r----- 1 root adm  3.9M Jul 26  2015 access.log.20.gz
-rw-r----- 1 root adm   20M Nov 30 00:12 access.log.2.gz
-rw-r----- 1 root adm   15M Nov 22 00:12 access.log.3.gz
-rw-r----- 1 root adm   16M Nov 16 00:12 access.log.4.gz
-rw-r----- 1 root adm   15M Nov  8 00:12 access.log.5.gz
-rw-r----- 1 root adm   13M Nov  1 00:12 access.log.6.gz
-rw-r----- 1 root adm   18M Oct 26 00:12 access.log.7.gz
-rw-r----- 1 root adm   16M Oct 18 00:12 access.log.8.gz
-rw-r----- 1 root adm   14M Oct 11 00:12 access.log.9.gz

It even says write in the output:

renaming /var/log/apache2/access.log to /var/log/apache2/access.log.1
creating new /var/log/apache2/access.log mode = 0640 uid = 0 gid = 4

So I am confused if this is apparently happening why nothing is actually happening? Can anyone identify the resolution to this problem?

Edit: In case it is relevant here is what my /etc/logrotate.d/apache2 file looks like:

/var/log/apache2/*.log {
        weekly
        missingok
        rotate 52
        compress
        delaycompress
        notifempty
        create 640 root adm
        sharedscripts
        postrotate
                if /etc/init.d/apache2 status > /dev/null ; then \
                    /etc/init.d/apache2 reload > /dev/null; \
                fi;
        endscript
        prerotate
                if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
                        run-parts /etc/logrotate.d/httpd-prerotate; \
                fi; \
        endscript
}

When I manually ran logrotate via sudo logrotate /etc/logrotate.conf the log correctly rotated. It looks like above I had -d which was causing the log to not rotate when I tried it manually. It seems like this is now an issue to do with the cron, and logrotate possibly not actually being ran in the cron?

Looking in my cron folders I do have a logrotate script at /etc/cron.daily/logrotate that looks like this:

#!/bin/sh

# Clean non existent log file entries from status file
cd /var/lib/logrotate
test -e status || touch status
head -1 status > status.clean
sed 's/"//g' status | while read logfile date
do
    [ -e "$logfile" ] && echo "\"$logfile\" $date"
done >> status.clean
mv status.clean status

test -x /usr/sbin/logrotate || exit 0
/usr/sbin/logrotate /etc/logrotate.conf

It seems like this should mean logrotate is running daily so there should be no problem with it not running?

Answer 1

If you are basing your assertion that the logs are not rotating based on the time stamp displayed by the ls command your concern is unjustified.

When a file is renamed the actual inodes on disk do not get touched so the time stamps do not change. You are basically changing the table of contents of the directory that holds the log files rather than changing the pages of the book. Only the newly created file gets a new time stamp associated with it.

You can confirm this with

zcat access.log.2.gz | head
zcat access.log.2.gz | tail

to confirm the time stamps of the first and last log entry in each file. Alternatively you can tell logrotate to append a timestamp to each logfile instead of a number.

Add the following lines to /etc/logrotate.d/apache2

dateext
dateformat -%Y-%m-%d-%s

[UPDATE]

While my previous assertion is true it is dependent on the operating system, file system, and mount options used. I just checked some of my more current servers and the time stamps do align on a default Ubuntu 14.04LTS install.

Do you have sufficient free space:

df -h /var/log

Is the file system mounted read-only by chance? You can either try to write a file to it or check the output of this command. And look for a rw or ro in the output.

mount | grep df /var/log/apache2/ | tail -1 | cut -d' ' -f1

On my test system the above command produces this output. Its the first rw in the ()'s that indicates its read/write and not read-only.

/dev/xvda1 on / type ext4 (rw,noatime,errors=remount-ro,barrier=0)

The file sizes of the gz files are not surprising. Log files compress extremely well since they contain a lot of duplicate data.

Answer 2

It seems to me you're in a strange environment somehow or logrotate is not configured correctly -- can you post your config? That said, you have stated that you used the -d option, which means "Don't do anything" -- does it work without that option? In that case you may have an issue with your cron or cron job -- What does the cron line / job say?

Otherwise, what I would check is:

• Verify logrotate is configured correctly -- I don't recall offhand if it supports this, but perhaps it has an option (in the config -- see above for -d) for a "dry run" enabled (i.e. say what it would do, but not actually do it)?
• You and logrotate are looking at the same paths, e.g. is it possible someone misconfigured it and it is in a chroot?
• Is the mountpoint behaving correctly, e.g. if these files are being saved to a full disk or network share, perhaps some of the changes are being rejected; Can you manually effect some of the changes such as renaming files?