0

I am using CentOS Linux release 7.0.1406 (Core).

The last time I logged in to SSH of the server was on April 20. Everything was working fine.

Today I logged in once again to check if anything new in the error.log of my websites. I do it periodically. But today there was a surprise:

[root@myserver nginx]# ls -la
total 104840
drwx------ 2 nginx nginx     4096 Apr 30 03:19 .
drwxr-xr-x 7 root  root      4096 May  3 03:20 ..
-rw-r--r-- 1 web   nginx        0 Apr 30 03:19 access.log
-rw-r--r-- 1 root  root  17956729 Apr 30 03:19 access.log-20150430.gz
-rw-r--r-- 1 web   nginx        0 Apr 30 03:19 awstats.site1.net.access.log
-rw-r--r-- 1 root  root      5229 Apr  2 14:21 awstats.site1.net.access.log-20150430.gz
-rw-r--r-- 1 web   nginx        0 Apr 30 03:19 awstats.site1.net.error.log
-rw-r--r-- 1 root  root      4654 Apr  2 14:21 awstats.site1.net.error.log-20150430.gz
-rw-r--r-- 1 web   nginx        0 Apr 30 03:19 devel.site1.net.access.log
-rw-r--r-- 1 root  root     26082 Apr 20 21:12 devel.site1.net.access.log-20150430.gz
-rw-r--r-- 1 web   nginx        0 Apr 30 03:19 devel.site1.net.error.log
-rw-r--r-- 1 root  root     46743 Apr 20 21:14 devel.site1.net.error.log-20150430.gz
-rw-r--r-- 1 web   nginx        0 Apr 30 03:19 devel.site2.pl.access.log
-rw-r--r-- 1 root  root      1652 Apr 24 06:28 devel.site2.pl.access.log-20150430.gz
-rw-r--r-- 1 web   nginx        0 Apr 30 03:19 devel.site2.pl.error.log
-rw-r--r-- 1 root  root       237 Feb 28 21:32 devel.site2.pl.error.log-20150430.gz
-rw-r--r-- 1 web   nginx        0 Apr 30 03:19 error.log
-rw-r--r-- 1 root  root    596623 Apr 30 02:38 error.log-20150430.gz
-rw-r--r-- 1 web   nginx        0 Apr 30 03:19 site1.net.access.log
-rw-r--r-- 1 root  root  83764451 Apr 30 03:18 site1.net.access.log-20150430.gz
-rw-r--r-- 1 web   nginx        0 Apr 30 03:19 site1.net.error.log
-rw-r--r-- 1 root  root    147462 Apr 29 21:36 site1.net.error.log-20150430.gz
-rw-r--r-- 1 web   nginx        0 Apr 30 03:19 site3.com-access.log
-rw-r--r-- 1 root  root    177285 Apr 30 03:14 site3.com-access.log-20150430.gz
-rw-r--r-- 1 web   nginx        0 Apr 30 03:19 site3.com-error.log
-rw-r--r-- 1 root  root     27929 Apr 28 23:16 site3.com-error.log-20150430.gz
-rw-r--r-- 1 web   nginx        0 Apr 30 03:19 panel.site4.com-access.log
-rw-r--r-- 1 root  root      1963 Apr 25 22:22 panel.site4.com-access.log-20150430.gz
-rw-r--r-- 1 web   nginx        0 Apr 30 03:19 panel.site4.com-error.log
-rw-r--r-- 1 root  root       488 Apr 13 14:21 panel.site4.com-error.log-20150430.gz
-rw-r--r-- 1 web   nginx        0 Apr 30 03:19 site2.pl.access.log
-rw-r--r-- 1 root  root   4485845 Apr 30 03:12 site2.pl.access.log-20150430.gz
-rw-r--r-- 1 web   nginx        0 Apr 30 03:19 site2.pl.error.log
-rw-r--r-- 1 root  root     61613 Apr 30 01:36 site2.pl.error.log-20150430.gz

As you can see, the .log files were 0KB!!! But there was a plenty of data there. It just... flew away.

I also noticed that with last, there was a strange reboot I was not aware of:

reboot   system boot  2.6.32-042stab08 Wed Apr 29 20:41 - 15:09 (8+18:27)

Now I changed back the owner/group to nginx and it looks like the logs are once again populating.

EDIT:

Here is my nginx.conf:
user web;
worker_processes 2;
pid /var/run/nginx.pid;

events {
        worker_connections 768;
        multi_accept on;
}

http {
        rewrite_log off;
        ##
        # Basic Settings
        ##
        client_max_body_size 20m;
        sendfile off;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 65;
        types_hash_max_size 2048;
        server_tokens off;

        # server_names_hash_bucket_size 64;
        # server_name_in_redirect off;

#       log_format  main  '$remote_addr $host $remote_user [$time_local] "$request" '
#                          '$status $body_bytes_sent "$http_referer" "$request_time"';

        log_format main     '$remote_addr - $remote_user [$time_local] "$request" '
                            '$status $body_bytes_sent "$http_referer" '
                            '"$http_user_agent" "$http_x_forwarded_for"';

        include /etc/nginx/mime.types;
        default_type application/octet-stream;



        proxy_buffer_size 128k;
        proxy_buffers 4 256k;
        proxy_busy_buffers_size 256k;

        fastcgi_buffer_size  16k;
        fastcgi_buffers      16  16k;
        ##
        # Logging Settings
        ##

        access_log /var/log/nginx/access.log main;
        error_log /var/log/nginx/error.log;

        ##
        # Gzip Settings
        ##


        gzip             on;
        gzip_disable "msie6";
        gzip_min_length  1000;
        gzip_proxied     expired no-cache no-store private auth;
        gzip_types       text/plain application/xml text/css text/js text/xml application/x-javascript text/javascript application/json application/xml+rss;

        ##
        # nginx-naxsi config
        ##
        # Uncomment it if you installed nginx-naxsi
        ##
        #include /etc/nginx/naxsi_core.rules;

        ##
        # nginx-passenger config
        ##
        # Uncomment it if you installed nginx-passenger
        ##

        #passenger_root /usr;
        #passenger_ruby /usr/bin/ruby;

        ##
        # Virtual Host Configs
        ##

        include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/sites-enabled/*;
}

Here is the output of: ps axu | grep log

root        86  0.0  0.0  34636   848 ?        Ss   Apr29   0:05 /usr/lib/systemd/systemd-logind
root       541  0.0  0.0   9512   588 ?        S    Apr29   0:01 dovecot/log
mysql      593  0.6  5.7 1675596 179496 ?      Sl   Apr29  89:36 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=/var/log/mariadb/mariadb.log --pid-file=/var/run/mariadb/mariadb.pid --socket=/var/lib/mysql/mysql.sock
root     30100  0.0  0.0   8988   900 pts/1    S+   20:31   0:00 grep --color=auto log

I have few questions:

  1. I don't remember if there were .gz files. But now there are. How/where can I check if there is some rule somewhere that says that it should gzip each logfile?
  2. What do you think happened? Is there anything else I can check to find the root cause of that issue
  3. Is there a way to prevent such things happening in future?
  4. Is there a way to recover the logs that disappeared?
6
  • What else are you running on this VPS? – Michael Hampton May 8 '15 at 15:41
  • it's just a webserver with mail and ftp... php-fpm, nginx, dovecot – Pikk May 8 '15 at 17:01
  • That's all? There's nothing else? There must be something. – Michael Hampton May 8 '15 at 17:01
  • well there is sshd, crond, systemd-journald, systemd-udevd, systemd-logind, /bin/dbus-daemon, agetty, mysqld, tlsmgr, but I don't think they are relevant. I noticed in the Facebook page of the VPS provider that on April 30 and May 1 some people were complaining about some outages.... I am wondering if some problem in the provider side, or some crash in OpenVZ crashed or corrupted or changes something in my VPS instance... – Pikk May 8 '15 at 17:06
  • Well then. Log rotation is normal, but not switching to log to the new logs is not. I think you should look at how you installed and configured nginx. – Michael Hampton May 8 '15 at 17:08
0

Gzipping depends on the log rotator¹. Though this link is for Ubuntu, /etc/logrotate.d/nginx also applies to CentOS. To see what's probably handling logging:

   $ ps axu | grep log

One of the processes that appear is likely what you want (for example rsyslogd).

What happened? Led by your comments, vendor restart. To confirm:

Login information (service providers may come in as root with a public key) may appear in /var/log files, such as: secure, messages, btmp (access with lastb), lastlog (assuming you're using a different account, it will show the last login for theirs, otherwise you've overwritten it) and account history files may show actions: (~/.bash_history and other ~/.files).

Prevention depends on cause. If the vendor did it, probably not (but you can ask how to get warned in the future).

Recovery is also "probably not". I've seen nginx unable to write its logs when they're not rotated properly (log rotation should include sending the USR1 signal to nginx). On startup it might be an initial permissions problem. As far as I know, nothing by default buffers unwritable logs, they're just gone.

¹ For easy access to the gzipped files:

    $ zcat each_file      # to see one at a time
    $ less *.gz           # less will zcat for you. :n and :p for next/previous file
    $ zcat *.gz | less    # to see all files in one big stream
3
  • Thanks, can I edited the first post... can you please take a look at the output of $ ps axu | grep log ? – Pikk May 8 '15 at 18:35
  • A log manager isn't apparent in that output. I just found 'journalctl' from: unix.stackexchange.com/questions/88744/… - does that show recent data? – ǝɲǝɲbρɯͽ May 8 '15 at 19:00
  • Yes, journalctl logged and is still logging everything but the webserver stuff. I think that journalctl isn't supposed to log www access/error logs. – Pikk May 8 '15 at 19:05
0

Here is the problem:

user web;

Nginx expects to run as user nginx, and all the associated scripts like logrotate expect this as well. This should never be changed unless you know exactly what you're doing and every possible implication. Revert it to the default:

user nginx;

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.