NGINX service rotate doesn't use new log files

Question

I'm having issues with any commands which tell nginx to use new log files. I'm using Ubuntu 18.04, nginx/1.14.0, logrotate 3.11.0

If I create new access.log or error.log files in /var/log/nginx, either manually or via logrotate, they do not get used and instead the service uses the old log files (which I have renamed to access.log.1 for this test, simulating what logrotate does.)

I have tried the following commands (separately). None of them produce any error message, they all produce expected output. But nginx refuses to stop using the old logs.

service nginx rotate

invoke-rc.d nginx rotate

kill -USR1 `cat /var/run/nginx.pid`

I have also verified that the above .pid file is in the correct place.

The only way I have been able to get log rotation working is via a service nginx reload, which does the job, but also reloads config files. I know there is no downtime with a reload, but I would still prefer to reload as little as possible which is why I want to get service nginx rotate working.

I am almost certain it is due to permission issues in /var/log. Recently we have set up a cronjob to ensure logfiles have secure permissions. This was for an audit, as the pentest company suggested various security measures in regards to logging. The cronjob we set up runs on boot:

#!/bin/bash
  
setfacl -Rm u::rwx,g::r--,o::--- /var/log
find /var/log -type f -exec chmod g-wx,o-rwx "{}" + -o -type d -exec chmod g-w,o-rwx "{}" +
chmod g+wx /var/log

chown -R www-data:adm /var/log/nginx

Here is the permissions of relevant directories and files after running the cronjob:

The /var/log directory itself:

drwxrwx--- 14 root syslog  4096 Aug 27 10:01 log

The /var/log/nginx directory itself:

drwxr-----  2 www-data  adm               4096 Aug 24 02:25  nginx

And the contents of /var/log/nginx (We use custom named logs in our nginx conf):

-rwxr----- 1 www-data adm     0 Aug 24 02:24 access.log
-rwxr----- 1 www-data adm   108 Aug 24 02:24 error.log
-rwxr----- 1 www-data adm 49317 Aug 27 10:11 x3nr0s.access.log
-rwxr----- 1 www-data adm   798 Aug 27 10:02 x3nr0s.error.log

If we run logrotate --force /etc/logrotate.d/nginx -v (verbosely), or even a manual touch to create new files, they are created with 640 permissions (as per logrotate's config file). From what I have read, 640 is enough:

-rwxr----- 1 www-data adm     0 Aug 24 02:24 access.log
-rw-r----- 1 www-data adm     0 Aug 27 10:13 error.log
-rwxr----- 1 www-data adm  1972 Aug 27 10:13 error.log.1
-rw-r----- 1 www-data adm     0 Aug 27 10:13 x3nr0s.access.log
-rwxr----- 1 www-data adm 51521 Aug 27 10:13 x3nr0s.access.log.1
-rw-r----- 1 www-data adm     0 Aug 27 10:13 x3nr0s.error.log
-rwxr----- 1 www-data adm   798 Aug 27 10:02 x3nr0s.error.log.1

As you can see, the new files stay empty, and logging continues to the old file. I've also verified the verbose logrotate output, and everything seems to be working OK in regards to the postrotate section. (Which runs invoke-rc.d nginx rotate. As I mentioned earlier, this command doesn't seem to rotate anything...)

As a last test, I tried giving user execute permissions to the new files, and did a service nginx rotate. Still, nginx uses the old files. Some other answers mention to check if the disk space is full. It is not.

Would appreciate assistance with this! Thanks.

Further Info

Here is my /etc/logrotate.d/nginx config:

/var/log/nginx/*.log {
        daily
        missingok
        rotate 14
        compress
        delaycompress
        notifempty
        create 0640 www-data adm
        sharedscripts
        prerotate
                if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
                        run-parts /etc/logrotate.d/httpd-prerotate; \
                fi \
        endscript
        postrotate
                invoke-rc.d nginx rotate >/dev/null 2>&1
        endscript
}

As mentioned above, the only way I have got logrotate and nginx to work correctly with the new log files is replacing the postrotate section with service nginx reload.

Here is the output of ps -ef | grep nginx:

root      1367     1  0 10:45 ?        00:00:00 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
www-data  1368  1367  0 10:45 ?        00:00:00 nginx: worker process
www-data  1369  1367  0 10:45 ?        00:00:00 nginx: worker process
www-data  1370  1367  0 10:45 ?        00:00:00 nginx: worker process
www-data  1371  1367  0 10:45 ?        00:00:00 nginx: worker process
root     15247 14835  0 11:19 pts/0    00:00:00 grep --color=auto nginx

Here is the getfacl on /var/log:

# file: var/log
# owner: root
# group: syslog
user::rwx
group::rwx
other::---

And here is the getfacl on /var/log/nginx:

# file: var/log/nginx
# owner: www-data
# group: adm
user::rwx
group::r--
other::---

Answer 1

The /etc/logrotate.d/nginx is the default one and should work.

However, changing the postrotate.d command, to force nginx to reload its config (and use the new log file)

  service nginx reload >/dev/null 2>&1

might be a quick fix.

Often when a usually-working service cannot create / open / rename / change a file, access rights are involved.

Here the accesses have been changed (for the sake of security), but the default acl should be working / safe as well, without involving setfacl which is powerful, but doesn't appear immediately when using the default ls options.

The default acl is, for /var/log

drwxrwxr-x 18 root syslog 4096 Aug 27 07:25 /var/log/

and the default nginx (actually) is

drwxr-x--- 2 www-data adm 4096 Aug 27 07:25 /var/log/nginx/

nginx is fine, /var/log could be a bit tighter

drwxrwx--x 18 root syslog 4096 Aug 27 07:25 /var/log/

that allow others to visit the dir (and below) but prevent them from listing the content.

As for the facl, the getfacl command will list what are the actual added ACLs for /var/log, /var/log/nginx and /var/log/nginx/*, that might reveal a problem.

By the way, do also a

dpkg-statoverride --list

to check what ACLs the installer will set after an update or install, and maybe change or add a line if necessary (man dpkg-statoverride)