logrotate fails to instruct Apache to reopen logs on Amazon Linux

Question

I have very good experience with logrotate normally, but on a recent installation using "Amazon Linux" (CentOS 6 deployed by Amazon) on EC2, logrotate fails to cause Apache to reopen its logs.

The logrotate configuration for Apache is slightly modified from the default - because of the high traffic on the site, the minimal requirements for keeping old logs and the small amount of space available on the EC2 instance store, we rotate the logs daily and only keep 2 days old files:

# cat /etc/logrotate.d/httpd 
/var/log/httpd/*log {
    missingok
    notifempty
    sharedscripts
    daily
    rotate 2
    compress
    postrotate
        /sbin/service httpd reload > /dev/null 2>/dev/null || true
    endscript
}

The rotation itself works fine and the old logs are compressed. But Apache doesn't actually reopen the log files after logrotate deletes the old files, and as a result the old files are not actually removed and Apache keeps piping data into them - eating up the available space and actually losing the log data because its not accessible.

If I run the postrotate script manually from the command line (with sudo or as root) then Apache reopens the log and I get the expected behavior.

There is nothing interesting in the logs (the cron logs says "logrotate started" followed by "logrotate finished") and the system mail has no notifications about problems with the cron job.

This has been going on for a couple of weeks now (since we moved the production server from a Debian based install to an "Amazon Linux" based install) and I'm now quite clueless as to what to do to track down the problem. Any help will be appreciated.

Answer 1

A couple of suggestions:

• Add a logger line before and after your postrotate service call to determine whether or not the postrotate section is being activated.

• I have not worked with service much, but if it has any knobs for increasing verbosity or logging to a file, I'd remove the >/dev/null redirects (so that the verbose output is actually sent!) and try any verbosity/logging knobs available.

• You mention that the logs aren't rotated, but are there other signs that Apache was successfully reloaded? To determine this, leave out the log-rotation steps (so that your logs aren't munched up), but leave the restart in place. If some evidence of attempted restart isn't showing up in the Apache logs, then Apache isn't really being restarted. The relevant logs lines usually include something like one of these:

[notice] Graceful restart requested, doing restart

[notice] Apache configured -- resuming normal operations

• I am not familiar with Amazon's implementation of CentOS, but if apachectl is available, you could try temporarily replacing your service call with an apachectl graceful, to see if that works. (I wouldn't leave it that way, but I'd try it just to add to your diagnostic knowledge).

Answer 2

The problem was that the logrotate configuration file for Apache did not run at all, because it had DOS new-line characters in it ( CR LF ). When that happens, logrotate annoyingly just skips the file without reporting any problem in the error log.

I discovered this by running logrotate manually on the httpd.conf file with debug mode (-d).

Fixing the new lines fixed the problem.