nginx stop logging after crontab modified the log file

Question

I have a web server(django) running on nginx + gunicorn. nginx writes its regular logs to:

~/myproject/log/access.log

and it writes errors to:

~/myproject/log/error.log

it works pretty well for a few months.

Recently I found that I have to find a way to automatically delete these logs before they occupy all of the disk space, so I wrote a script to delete parts of the file if it exceeds more than x lines, and I let crontab to run the script every 24 hours.

Then I noticed that the nginx stop logging each time the crontab has executed my script. I can recover it by restarting nginx, but it will stop logging again after the crontab doing its job.

Here is my script:

LOG_PATH="/home/ubuntu/myproject/log/"
MAX_LINE_COUNT="400000"
NOW=$(TZ='Asia/Taipei' date +"%Y-%m-%d %H:%M")

echo "$NOW (GMT+8) start truncate_logs.sh"

for f in "$LOG_PATH"*; do
    line_count=`wc -l < "$f"`
    echo "$f has $line_count lines"

    if [ "$line_count" -gt "$MAX_LINE_COUNT" ]
        then
            lines_to_truncate=$((line_count-MAX_LINE_COUNT))
            sed -i '1,'"$lines_to_truncate"'d' "$f"
            echo "truncated $lines_to_truncate lines"
        else
            echo "pass"
    fi
done

I wonder if nginx and crontab both access the file at the same time and it breaks the nginx's logging behavior?

In addition, when nginx stop writing logs in "access.log", it still writes errors in "error.log". The "error.log" file always has lesser lines so my script has never modified the file. It's one of the reason that I suspect the modification of my script on "access.log" has changed nginx's behavior.

Answer 1

You should use a proper nginx logrotate script on this matter.

Here is an example:

"/var/log/nginx/*.log" {
        daily
        rotate 7
        size 100M
        dateext
        dateformat -%Y%m%d-%s
        compress
        delaycompress
        missingok
        notifempty
        create 0640 www-data adm
        sharedscripts
        prerotate
                if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
                        run-parts /etc/logrotate.d/httpd-prerotate; \
                fi \
        endscript
        postrotate
                [ ! -f /run/nginx.pid ] || kill -USR1 `cat /run/nginx.pid`
        endscript
}

You can also enable lorotation to check the file size each hour instead of daily.

sudo ln -s /etc/cron.daily/logrotate /etc/cron.hourly/logrotate

The examples are from ubuntu server 14.04, but should be compatible to other *Nix systems.