Apache crash every day

Question

My I have a clean Debian server with apache only, I have 1 basic site with no special options. Everything is working fine but everyday at the same hour, Apache will crash and won't start, I have to stop the site and start apache manually. Log from /var/log/syslog:

Aug 8 06:25:01 localhost CRON[8096]: (root) CMD (test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ))

Aug 8 06:25:17 localhost systemd[1]: Reloading LSB: Apache2 web server.

Aug 8 06:25:18 localhost apache2[8220]: Reloading web server: apache2.

Aug 8 06:25:18 localhost systemd[1]: Reloaded LSB: Apache2 web server.

Aug 8 06:25:18 localhost apache2[8237]: Stopping web server: apache2.

Aug 8 06:25:18 localhost rsyslogd: [origin software="rsyslogd" swVersion="8.4.2" x-pid="550" x-info="http://www.rsyslog.com"] rsyslogd was HUPed

Log from /var/log/apache/error.log:

[Wed Aug 08 06:25:18.467344 2018] [mpm_event:notice] [pid 7493:tid 139910286395264] AH00493: SIGUSR1 received. Doing graceful restart AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1. Set the 'ServerName' directive globally to suppress this message

[Wed Aug 08 06:25:18.521093 2018] [core:crit] [pid 7493:tid 139910286395264] (22)Invalid argument: AH00069: make_sock: for address [::]:80, apr_socket_opt_set: (IPV6_V6ONLY) (98)Address already in use: AH00072: make_sock: could not bind to address 0.0.0.0:80 [Wed Aug 08 06:25:18.521141 2018]

[mpm_event:alert] [pid 7493:tid 139910286395264] no listening sockets available, shutting down

[Wed Aug 08 06:25:18.521149 2018] [:emerg] [pid 7493:tid 139910286395264] AH00019: Unable to open logs, exiting

Answer 1

From the timing and log messages, I suspect the cause is logrotate.

There will be a file on your system called /etc/logrotate.d/apache2, which configures the behavior of logrotate with regards to your apache web server.

The rsyslog version 8.4.2 indicates that you're using Debian Jessie. Here's what it looks like on a Jessie system of mine:

/var/log/apache2/*.log {
        daily
        missingok
        rotate 14
        compress
        delaycompress
        notifempty
        create 640 root adm
        sharedscripts
        postrotate
                if /etc/init.d/apache2 status > /dev/null ; then \
                    /etc/init.d/apache2 reload > /dev/null; \
                fi;
        endscript
        prerotate
                if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
                        run-parts /etc/logrotate.d/httpd-prerotate; \
                fi; \
        endscript
}

The interesting part here is the postrotate. It calls the apache2 init script with the reload option.

The point of this is to have apache release the file handles of the log files, so that it will start logging to fresh files, after the old ones have been moved away.

If I were you I would manually run the postrotate action, which is presumably the same apache reload, during a maintenance window to see if it also crashes apache.

If it does, you'll want to find out why it crashes, or simply find an alternative postrotate action.

Some reasonable postrotate options for Apache2 are /usr/sbin/apachectl graceful or /usr/sbin/apachectl restart.

I also really like copytruncate instead of using a postrotate action. See man 8 logrotate for more info on copytruncate).

You could play around with these and see which works for you.