4

I have a production system on which there are multiple nginx apps hosted. those are nginx applications servers. Now for each of these nginx app servers a logrotate config file is configured in /etc/logrotate.d/ folder. one of the example of logrotate config file is given below.

/var/www/example/current/log/example.log {
  daily
  missingok
  rotate 1
  delaycompress
  compress
  notifempty
  copytruncate
}

I checked following Logrotate not working and checked my config and didn't find any duplicate entries for application servers or nginx as well. below is the definition of my nginx logrotate file.

/var/log/nginx/*.log {
    daily
    missingok
    rotate 3
  compress
  delaycompress
    notifempty
    create 0640 www-data adm
    sharedscripts
    postrotate
    if [ -s /var/run/nginx.pid ]; then
      kill -USR1 `cat /var/run/nginx.pid`
    fi
    endscript
}

My logs are still getting written in *.log.1 and not in *.log Can anyone help with this? I am trying to understand what is wrong here. Below is the logrotate command output in debug mode.

logrotate --force -d /etc/logrotate.d/nginx
reading config file /etc/logrotate.d/nginx

Handling 1 logs

rotating pattern: /var/log/nginx/*.log  forced from command line (3 rotations)
empty log files are not rotated, old logs are removed
considering log /var/log/nginx/access.log
  log does not need rotating
considering log /var/log/nginx/elasticsearch.access.log
  log does not need rotating
considering log /var/log/nginx/error.log
  log does not need rotating
considering log /var/log/nginx/grafana.access.log
  log does not need rotating
considering log /var/log/nginx/example-admin.access.log
  log needs rotating
considering log /var/log/nginx/example-admin.error.log
  log does not need rotating
considering log /var/log/nginx/example-administrators.access.log
  log needs rotating
considering log /var/log/nginx/example-administrators.error.log
  log does not need rotating
considering log /var/log/nginx/example-alot.access.log
  log does not need rotating
considering log /var/log/nginx/example-alot.error.log
  log does not need rotating
considering log /var/log/nginx/example-api_clients.access.log
  log does not need rotating
considering log /var/log/nginx/example-api_clients.error.log
  log does not need rotating
considering log /var/log/nginx/example-assets.access.log
  log does not need rotating
considering log /var/log/nginx/example-backgrounds.access.log
  log does not need rotating
considering log /var/log/nginx/example-backgrounds.error.log
  log does not need rotating
considering log /var/log/nginx/example-blog.access.log
  log does not need rotating
considering log /var/log/nginx/example-blog.error.log
  log does not need rotating
considering log /var/log/nginx/example-blog-service.access.log
  log does not need rotating
considering log /var/log/nginx/example-blog-service.error.log
  log does not need rotating
considering log /var/log/nginx/example-boards.access.log
  log does not need rotating
considering log /var/log/nginx/example-boards.error.log
  log does not need rotating
considering log /var/log/nginx/example-bolt.access.log
  log does not need rotating
considering log /var/log/nginx/example-bolt.error.log
  log does not need rotating
considering log /var/log/nginx/example-bonobo.access.log
  log does not need rotating
considering log /var/log/nginx/example-bonobo.error.log
  log does not need rotating
considering log /var/log/nginx/example-boss.access.log
  log does not need rotating
considering log /var/log/nginx/example-boss.error.log
  log does not need rotating
considering log /var/log/nginx/example-callbacks.access.log
  log does not need rotating
considering log /var/log/nginx/example-callbacks.error.log
  log does not need rotating
considering log /var/log/nginx/example-cashier.access.log
  log does not need rotating
considering log /var/log/nginx/example-cashier.error.log
  log does not need rotating
considering log /var/log/nginx/example-dashboard-old.access.log
  log does not need rotating
considering log /var/log/nginx/example-dcl_shipments.access.log
  log does not need rotating
considering log /var/log/nginx/example-dcl_shipments.error.log
  log does not need rotating
considering log /var/log/nginx/example-device-registry.access.log
  log does not need rotating
considering log /var/log/nginx/example-device-registry.error.log
  log does not need rotating
considering log /var/log/nginx/example-docs.access.log
  log does not need rotating
considering log /var/log/nginx/example-exports.access.log
  log does not need rotating
considering log /var/log/nginx/example-exports.error.log
  log does not need rotating
considering log /var/log/nginx/example-fake-api.access.log
  log does not need rotating
considering log /var/log/nginx/example-fake-api.error.log
  log does not need rotating
considering log /var/log/nginx/example-fashion-police.access.log
  log does not need rotating
considering log /var/log/nginx/example-fashion-police.error.log
  log does not need rotating
considering log /var/log/nginx/example-flight-recorder.access.log
  log does not need rotating
considering log /var/log/nginx/example-flight-recorder.error.log
  log does not need rotating
considering log /var/log/nginx/example-gone.access.log
  log does not need rotating
considering log /var/log/nginx/example-gone.error.log
  log does not need rotating
considering log /var/log/nginx/example-go-portal.access.log
  log does not need rotating
considering log /var/log/nginx/example-go-portal.error.log
  log does not need rotating
considering log /var/log/nginx/example-help-service.access.log
  log does not need rotating
considering log /var/log/nginx/example-help-service.error.log
  log does not need rotating
considering log /var/log/nginx/example-kfc.access.log
  log does not need rotating
considering log /var/log/nginx/example-kfc.error.log
  log does not need rotating
considering log /var/log/nginx/example-kredits.access.log
  log does not need rotating
considering log /var/log/nginx/example-kredits.error.log
  log does not need rotating
considering log /var/log/nginx/example-login.access.log
  log does not need rotating
considering log /var/log/nginx/example-login.error.log
  log does not need rotating
considering log /var/log/nginx/example-manual-activations.access.log
  log does not need rotating
considering log /var/log/nginx/example-manual-activations.error.log
  log does not need rotating
considering log /var/log/nginx/example-maps.access.log
  log does not need rotating
considering log /var/log/nginx/example-mobile-api.access.log
  log does not need rotating
considering log /var/log/nginx/example-mobile-api.error.log
  log does not need rotating
considering log /var/log/nginx/example-mqttproxy.access.log
  log does not need rotating
considering log /var/log/nginx/example-mqttproxy.error.log
  log does not need rotating
considering log /var/log/nginx/example-network-coverage.access.log
  log does not need rotating
considering log /var/log/nginx/example-network-coverage.error.log
  log does not need rotating
considering log /var/log/nginx/example-package_tracker.access.log
  log does not need rotating
considering log /var/log/nginx/example-package_tracker.error.log
  log does not need rotating
considering log /var/log/nginx/example-people.access.log
  log needs rotating
considering log /var/log/nginx/example-people.error.log
  log does not need rotating
considering log /var/log/nginx/example-postman-pat.access.log
  log does not need rotating
considering log /var/log/nginx/example-postman-pat.error.log
  log does not need rotating
considering log /var/log/nginx/example-premium.access.log
  log does not need rotating
considering log /var/log/nginx/example-premium.error.log
  log does not need rotating
considering log /var/log/nginx/example-push-notifications.access.log
  log does not need rotating
considering log /var/log/nginx/example-push-notifications.error.log
  log does not need rotating
considering log /var/log/nginx/example-referrific.access.log
  log does not need rotating
considering log /var/log/nginx/example-referrific.error.log
  log does not need rotating
considering log /var/log/nginx/example-services.access.log
  log does not need rotating
considering log /var/log/nginx/example-services.error.log
  log does not need rotating
considering log /var/log/nginx/example-store.access.log
  log does not need rotating
considering log /var/log/nginx/example-store.error.log
  log does not need rotating
considering log /var/log/nginx/example-subs.access.log
  log does not need rotating
considering log /var/log/nginx/example-subs.error.log
  log does not need rotating
considering log /var/log/nginx/example-updates.access.log
  log does not need rotating
considering log /var/log/nginx/example-updates.error.log
  log does not need rotating
considering log /var/log/nginx/example-website.access.log
  log does not need rotating
considering log /var/log/nginx/example-wildcard-redirect.access.log
  log does not need rotating
considering log /var/log/nginx/example-zendesk-proxy.access.log
  log does not need rotating
considering log /var/log/nginx/example-zendesk-proxy.error.log
  log does not need rotating
considering log /var/log/nginx/kibana.access.log
  log does not need rotating
considering log /var/log/nginx/prometheus.access.log
  log does not need rotating
considering log /var/log/nginx/prometheus_alertmanager.access.log
  log does not need rotating
considering log /var/log/nginx/prometheus_statsd_bridge.access.log
  log does not need rotating
rotating log /var/log/nginx/example-admin.access.log, log->rotateCount is 3
dateext suffix '-20170104'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
compressing log with: /bin/gzip
renaming /var/log/nginx/example-admin.access.log.3.gz to /var/log/nginx/example-admin.access.log.4.gz (rotatecount 3, logstart 1, i 3), 
renaming /var/log/nginx/example-admin.access.log.2.gz to /var/log/nginx/example-admin.access.log.3.gz (rotatecount 3, logstart 1, i 2), 
renaming /var/log/nginx/example-admin.access.log.1.gz to /var/log/nginx/example-admin.access.log.2.gz (rotatecount 3, logstart 1, i 1), 
renaming /var/log/nginx/example-admin.access.log.0.gz to /var/log/nginx/example-admin.access.log.1.gz (rotatecount 3, logstart 1, i 0), 
rotating log /var/log/nginx/example-administrators.access.log, log->rotateCount is 3
dateext suffix '-20170104'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
compressing log with: /bin/gzip
renaming /var/log/nginx/example-administrators.access.log.3.gz to /var/log/nginx/example-administrators.access.log.4.gz (rotatecount 3, logstart 1, i 3), 
renaming /var/log/nginx/example-administrators.access.log.2.gz to /var/log/nginx/example-administrators.access.log.3.gz (rotatecount 3, logstart 1, i 2), 
renaming /var/log/nginx/example-administrators.access.log.1.gz to /var/log/nginx/example-administrators.access.log.2.gz (rotatecount 3, logstart 1, i 1), 
renaming /var/log/nginx/example-administrators.access.log.0.gz to /var/log/nginx/example-administrators.access.log.1.gz (rotatecount 3, logstart 1, i 0), 
rotating log /var/log/nginx/example-people.access.log, log->rotateCount is 3
dateext suffix '-20170104'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
compressing log with: /bin/gzip
renaming /var/log/nginx/example-people.access.log.3.gz to /var/log/nginx/example-people.access.log.4.gz (rotatecount 3, logstart 1, i 3), 
renaming /var/log/nginx/example-people.access.log.2.gz to /var/log/nginx/example-people.access.log.3.gz (rotatecount 3, logstart 1, i 2), 
renaming /var/log/nginx/example-people.access.log.1.gz to /var/log/nginx/example-people.access.log.2.gz (rotatecount 3, logstart 1, i 1), 
renaming /var/log/nginx/example-people.access.log.0.gz to /var/log/nginx/example-people.access.log.1.gz (rotatecount 3, logstart 1, i 0), 
renaming /var/log/nginx/example-admin.access.log to /var/log/nginx/example-admin.access.log.1
creating new /var/log/nginx/example-admin.access.log mode = 0640 uid = 33 gid = 4
renaming /var/log/nginx/example-administrators.access.log to /var/log/nginx/example-administrators.access.log.1
creating new /var/log/nginx/example-administrators.access.log mode = 0640 uid = 33 gid = 4
renaming /var/log/nginx/example-people.access.log to /var/log/nginx/example-people.access.log.1
creating new /var/log/nginx/example-people.access.log mode = 0640 uid = 33 gid = 4
running postrotate script
running script with arg /var/log/nginx/*.log : "
    if [ -s /var/run/nginx.pid ]; then
      kill -USR1 `cat /var/run/nginx.pid`
    fi
"
removing old log /var/log/nginx/example-admin.access.log.4.gz
error: error opening /var/log/nginx/example-admin.access.log.4.gz: No such file or directory

And when i do force logrotate, I end up having errors below.

sudo logrotate -f /etc/logrotate.conf
logrotate_script: 2: logrotate_script: /etc/init.d/example-admin: not found
error: error running non-shared postrotate script for /var/www/admin/current/log/example-admin.log of '/var/www/admin/current/log/example-admin.log '
logrotate_script: 2: logrotate_script: /etc/init.d/example-administrators: not found
error: error running non-shared postrotate script for /var/www/administrators/current/log/example-administrators.log of '/var/www/administrators/current/log/example-administrators.log '
logrotate_script: 2: logrotate_script: /etc/init.d/example-alot: not found
error: error running non-shared postrotate script for /var/www/alot/current/log/example-alot.log of '/var/www/alot/current/log/example-alot.log '
logrotate_script: 2: logrotate_script: /etc/init.d/example-bonobo: not found
error: error running non-shared postrotate script for /var/www/bonobo/current/log/example-bonobo.log of '/var/www/bonobo/current/log/example-bonobo.log '
logrotate_script: 2: logrotate_script: /etc/init.d/example-cashier: not found
error: error running non-shared postrotate script for /var/www/cashier/current/log/example-cashier.log of '/var/www/cashier/current/log/example-cashier.log '
logrotate_script: 2: logrotate_script: /etc/init.d/example-dcl-shipments: not found
error: error running non-shared postrotate script for /var/www/dcl_shipments/current/log/example-dcl_shipments.log of '/var/www/dcl_shipments/current/log/example-dcl_shipments.log '
logrotate_script: 2: logrotate_script: /etc/init.d/example-people: not found
error: error running non-shared postrotate script for /var/www/people/current/log/example-people.log of '/var/www/people/current/log/example-people.log '
logrotate_script: 2: logrotate_script: /etc/init.d/example-store: not found
error: error running non-shared postrotate script for /var/www/store/current/log/example-store.log of '/var/www/store/current/log/example-store.log '
logrotate_script: 2: logrotate_script: /etc/init.d/example-storefront: not found
error: error running non-shared postrotate script for /var/www/storefront/current/log/example-storefront.log of '/var/www/storefront/current/log/example-storefront.log '
logrotate_script: 2: logrotate_script: /etc/init.d/example-subs: not found
error: error running non-shared postrotate script for /var/www/subs/current/log/example-subs.log of '/var/www/subs/current/log/example-subs.log '

Below is the logrotate command ouput in debug mode.

sudo logrotate -dv /etc/logrotate.d/example-alot
reading config file /etc/logrotate.d/example-alot

Handling 1 logs

rotating pattern: /var/www/alot/current/log/example-alot.log  after 1 days (1 rotations)
empty log files are not rotated, old logs are removed
considering log /var/www/alot/current/log/example-alot.log
  log needs rotating
rotating log /var/www/alot/current/log/example-alot.log, log->rotateCount is 1
dateext suffix '-20170105'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
compressing log with: /bin/gzip
renaming /var/www/alot/current/log/example-alot.log.1.gz to /var/www/alot/current/log/example-alot.log.2.gz (rotatecount 1, logstart 1, i 1),
renaming /var/www/alot/current/log/example-alot.log.0.gz to /var/www/alot/current/log/example-alot.log.1.gz (rotatecount 1, logstart 1, i 0),
copying /var/www/alot/current/log/example-alot.log to /var/www/alot/current/log/example-alot.log.1
truncating /var/www/alot/current/log/example-alot.log
running postrotate script
running script with arg /var/www/alot/current/log/example-alot.log: "
      echo "Appending to" example-alot > /var/www/alot/current/log/example-alot.log
      /etc/init.d/example-alot reload >> /dev/null
"
removing old log /var/www/alot/current/log/example-alot.log.2.gz


sudo logrotate -dv /etc/logrotate.d/example-subs
reading config file /etc/logrotate.d/example-subs

Handling 1 logs

rotating pattern: /var/www/subs/current/log/example-subs.log  after 1 days (1 rotations)
empty log files are not rotated, old logs are removed
considering log /var/www/subs/current/log/example-subs.log
  log does not need rotating
1
  • I understand that there are no init.d scripts available for these services and hence it is not possible for logrotate to start services again. Jan 4, 2017 at 19:47

2 Answers 2

0

If nginx is writing to the old log file, this might happen because your postrotate action is not working.

Did you check that /var/run/nginx.pid is existing and contains the correct pid?

Moreover you could try

logrotate --force -d /etc/logrotate.d/nginx

and then check the output. It gives good indications.

2
  • Yes. File exist and it contains correct pid. cat /var/run/nginx.pid 4103 ssutar@example-all-roles:~$ ps aux | grep 4103 root 4103 0.0 0.0 101992 5224 ? Ss 14:03 0:00 nginx: master process /usr/sbin/nginx ssutar 25544 0.0 0.0 10460 940 pts/0 S+ 19:49 0:00 grep --color=auto 4103 Jan 3, 2017 at 19:50
  • And whats the output if you run logrotate in debug mode?
    – turbophi
    Jan 4, 2017 at 8:55
0

If you're using syslog or syslog-ng, you need to make sure something like this is part of the shared postrotate scripts:

sharedscripts
postrotate
/bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
endscript

syslog writes files to inodes, not file names, so if you don't bounce syslog, it continues to write to the same inode (even though the file name changed) HUPing the syslog process resets the inode to the correct name. It's the same reason that syslog fails to write to files that are non-existent. You have to touch the file before starting syslog so that the inode of the file can be determined.

Hope that helps

2
  • 1
    It would be more accurate to say that syslog writes to an already-opened file handle, and doesn't re-open its files unless / until you tell it to with a SIGHUP. I don't think you are right about the reason why syslog doesn't open a file that doesn't already exist.
    – mc0e
    May 29, 2021 at 8:43
  • True ... nodes just live below the file handle space (physical disk layer), I've just spent way too much time looking @ inode issues. i.e. why does the disk/directory say it cannot open another file even though there's a lot of disk space Answer: The disk directory is out of inodes because of the > 20 million < 1K files that were opened there Jun 16, 2021 at 17:03

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .