2

I'm having issues with any commands which tell nginx to use new log files. I'm using Ubuntu 18.04, nginx/1.14.0, logrotate 3.11.0

If I create new access.log or error.log files in /var/log/nginx, either manually or via logrotate, they do not get used and instead the service uses the old log files (which I have renamed to access.log.1 for this test, simulating what logrotate does.)

I have tried the following commands (separately). None of them produce any error message, they all produce expected output. But nginx refuses to stop using the old logs.

service nginx rotate

invoke-rc.d nginx rotate

kill -USR1 `cat /var/run/nginx.pid`

I have also verified that the above .pid file is in the correct place.

The only way I have been able to get log rotation working is via a service nginx reload, which does the job, but also reloads config files. I know there is no downtime with a reload, but I would still prefer to reload as little as possible which is why I want to get service nginx rotate working.

I am almost certain it is due to permission issues in /var/log. Recently we have set up a cronjob to ensure logfiles have secure permissions. This was for an audit, as the pentest company suggested various security measures in regards to logging. The cronjob we set up runs on boot:

#!/bin/bash
  
setfacl -Rm u::rwx,g::r--,o::--- /var/log
find /var/log -type f -exec chmod g-wx,o-rwx "{}" + -o -type d -exec chmod g-w,o-rwx "{}" +
chmod g+wx /var/log

chown -R www-data:adm /var/log/nginx

Here is the permissions of relevant directories and files after running the cronjob:

The /var/log directory itself:

drwxrwx--- 14 root syslog  4096 Aug 27 10:01 log

The /var/log/nginx directory itself:

drwxr-----  2 www-data  adm               4096 Aug 24 02:25  nginx

And the contents of /var/log/nginx (We use custom named logs in our nginx conf):

-rwxr----- 1 www-data adm     0 Aug 24 02:24 access.log
-rwxr----- 1 www-data adm   108 Aug 24 02:24 error.log
-rwxr----- 1 www-data adm 49317 Aug 27 10:11 x3nr0s.access.log
-rwxr----- 1 www-data adm   798 Aug 27 10:02 x3nr0s.error.log

If we run logrotate --force /etc/logrotate.d/nginx -v (verbosely), or even a manual touch to create new files, they are created with 640 permissions (as per logrotate's config file). From what I have read, 640 is enough:

-rwxr----- 1 www-data adm     0 Aug 24 02:24 access.log
-rw-r----- 1 www-data adm     0 Aug 27 10:13 error.log
-rwxr----- 1 www-data adm  1972 Aug 27 10:13 error.log.1
-rw-r----- 1 www-data adm     0 Aug 27 10:13 x3nr0s.access.log
-rwxr----- 1 www-data adm 51521 Aug 27 10:13 x3nr0s.access.log.1
-rw-r----- 1 www-data adm     0 Aug 27 10:13 x3nr0s.error.log
-rwxr----- 1 www-data adm   798 Aug 27 10:02 x3nr0s.error.log.1

As you can see, the new files stay empty, and logging continues to the old file. I've also verified the verbose logrotate output, and everything seems to be working OK in regards to the postrotate section. (Which runs invoke-rc.d nginx rotate. As I mentioned earlier, this command doesn't seem to rotate anything...)

As a last test, I tried giving user execute permissions to the new files, and did a service nginx rotate. Still, nginx uses the old files. Some other answers mention to check if the disk space is full. It is not.

Would appreciate assistance with this! Thanks.

Further Info

Here is my /etc/logrotate.d/nginx config:

/var/log/nginx/*.log {
        daily
        missingok
        rotate 14
        compress
        delaycompress
        notifempty
        create 0640 www-data adm
        sharedscripts
        prerotate
                if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
                        run-parts /etc/logrotate.d/httpd-prerotate; \
                fi \
        endscript
        postrotate
                invoke-rc.d nginx rotate >/dev/null 2>&1
        endscript
}

As mentioned above, the only way I have got logrotate and nginx to work correctly with the new log files is replacing the postrotate section with service nginx reload.

Here is the output of ps -ef | grep nginx:

root      1367     1  0 10:45 ?        00:00:00 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
www-data  1368  1367  0 10:45 ?        00:00:00 nginx: worker process
www-data  1369  1367  0 10:45 ?        00:00:00 nginx: worker process
www-data  1370  1367  0 10:45 ?        00:00:00 nginx: worker process
www-data  1371  1367  0 10:45 ?        00:00:00 nginx: worker process
root     15247 14835  0 11:19 pts/0    00:00:00 grep --color=auto nginx

Here is the getfacl on /var/log:

# file: var/log
# owner: root
# group: syslog
user::rwx
group::rwx
other::---

And here is the getfacl on /var/log/nginx:

# file: var/log/nginx
# owner: www-data
# group: adm
user::rwx
group::r--
other::---
  • @e2-e4 Updated the description with these outputs – x3nr0s Aug 27 at 11:20
  • You could try to replace the postrotate command with nginx -s reload >/dev/null 2>&1 and if that doesn't work service nginx reload >/dev/null 2>&1 that will force nginx to (1) reload its config or (2) restart. (2) is stronger, (1) is better. I'm running the exact same versions as you and don't have the problem. Maybe you changed something? The uids seem to match, did you use chattr? – e2-e4 Aug 27 at 11:48
  • Yes, nginx -s reload (or service nginx reload) both work fine in the postrotate section, and the logs rotate. But I wish to know exactly why the rotate option, either with invoke-rc.d or service, does not work. Even sending -USR1 to nginx pid doesn't work. As mentioned, I suspect it is something to do with my scheduled cronjob and related setfacl/chmod commands within it. I did not use chattr. – x3nr0s Aug 27 at 11:56
  • 1
    Ok, missed that part. Indeed, likely to be the culprit. a getfacl /var/log and nginx would be useful. The base acl rwxrwx--- for root syslog will not allow any other user going below... so the facl are adding some rights, but they need to be checked below. Why doesn't the crontab set the default acl (without facl which is harder to see/trace), eg rwxrwxr-x or at least rwxrwx--x and removing the facl, then nginx should be rwxr-x--x for www-data adm – e2-e4 Aug 27 at 12:09
  • 1
    There we go! In my cronjob, I changed the third line to chmod g+wx, o+x /var/log. The way I have it set up, the rest of the directories and files within /var/log still have no other permissions, but only the parent directory /var/log has execute for other. Logrotate + nginx started working after that. Thanks! Feel free to post your info as an answer and I'll accept it. Thanks for the help! – x3nr0s Aug 27 at 13:00
1

The /etc/logrotate.d/nginx is the default one and should work.

However, changing the postrotate.d command, to force nginx to reload its config (and use the new log file)

  service nginx reload >/dev/null 2>&1

might be a quick fix.

Often when a usually-working service cannot create / open / rename / change a file, access rights are involved.

Here the accesses have been changed (for the sake of security), but the default acl should be working / safe as well, without involving setfacl which is powerful, but doesn't appear immediately when using the default ls options.

The default acl is, for /var/log

drwxrwxr-x 18 root syslog 4096 Aug 27 07:25 /var/log/

and the default nginx (actually) is

drwxr-x--- 2 www-data adm 4096 Aug 27 07:25 /var/log/nginx/

nginx is fine, /var/log could be a bit tighter

drwxrwx--x 18 root syslog 4096 Aug 27 07:25 /var/log/

that allow others to visit the dir (and below) but prevent them from listing the content.

As for the facl, the getfacl command will list what are the actual added ACLs for /var/log, /var/log/nginx and /var/log/nginx/*, that might reveal a problem.

By the way, do also a

dpkg-statoverride --list

to check what ACLs the installer will set after an update or install, and maybe change or add a line if necessary (man dpkg-statoverride)

| improve this answer | |
  • 1
    For future readers -- the issue was simply that the parent /var/log directory itself did not have execute permissions for other. Once this was added, nginx could read the new files created by logrotate, which was using the default nginx config containing invoke-rc.d nginx rotate in the postrotate section. – x3nr0s Aug 27 at 13:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.