0

IMPORTANT NOTE: I no longer need to restart the servers(my 3 VPS!) to have the pid file back in the dir /run/nginx.pid

I just typed the command: sudo fuser -k 443 / tcp && service nginx start

solves the problem for 24h, as the pid returns to /run/nginx/ without having to restart the service, and is always uptime, but every day the PID disappears! even though a nginx process is in progress and the server always uptime!

so I can use service nginx start and the pid returns to appear the monit returns the monitor the nginx process and I came back to have control over the commands: service nginx status, service nginx stop etc...

Attention, I do not say that nginx stops working, the sites continue oline, I say that the file: /run/nginx.pid disappears every day, so I can no longer use the nginx related commands, even me enter the command

sudo fuser -k 443/tcp

the absence of the file /run/nginx.pid brings several problems, the worst of them is I can not use the commands, service nginx status, start, stop etc..

========================= I need the nginx pid to work with the monit, the pid is simply disappearing unexpectedly, I have to restart the vps to get it back! every night this is happening! but even if he disappears, he's still running! the online site! but obviously I'm unable to monitor status, restart, stop ... etc ... I need pid! when I try to see the status via terminal I have

Error output on the terminal commandline:

service nginx status
● nginx.service - A high performance web server and a reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: en
   Active: failed (Result: exit-code) since Sun 2017-11-05 06:25:07 WET; 1h 50mi
  Process: 5062 ExecStop=/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 -
  Process: 1527 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code
  Process: 1522 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process 
 Main PID: 1486 (code=exited, status=0/SUCCESS)

Nov 05 06:25:06 vps1 nginx[1527]: nginx: [emerg] bind() to 141.14.14.41:443 fail
Nov 05 06:25:06 vps1 nginx[1527]: nginx: [emerg] bind() to 141.14.14.41:80 faile
Nov 05 06:25:06 vps1 nginx[1527]: nginx: [emerg] bind() to 141.14.14.41:443 fail
Nov 05 06:25:07 vps1 nginx[1527]: nginx: [emerg] bind() to 141.14.14.41:80 faile
Nov 05 06:25:07 vps1 nginx[1527]: nginx: [emerg] bind() to 141.14.14.41:443 fail
Nov 05 06:25:07 vps1 nginx[1527]: nginx: [emerg] still could not bind()
Nov 05 06:25:07 vps1 systemd[1]: nginx.service: Control process exited, code=exi
Nov 05 06:25:07 vps1 systemd[1]: Failed to start A high performance web server a
Nov 05 06:25:07 vps1 systemd[1]: nginx.service: Unit entered failed state.
Nov 05 06:25:07 vps1 systemd[1]: nginx.service: Failed with result 'exit-code'.
lines 1-18/18 (END)...skipping...
● nginx.service - A high performance web server and a reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: en
   Active: failed (Result: exit-code) since Sun 2017-11-05 06:25:07 WET; 1h 50mi
  Process: 5062 ExecStop=/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 -
  Process: 1527 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code
  Process: 1522 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process 
 Main PID: 1486 (code=exited, status=0/SUCCESS)

Nov 05 06:25:06 vps1 nginx[1527]: nginx: [emerg] bind() to 141.14.14.41:443 fail
Nov 05 06:25:06 vps1 nginx[1527]: nginx: [emerg] bind() to 141.14.14.41:80 faile
Nov 05 06:25:06 vps1 nginx[1527]: nginx: [emerg] bind() to 141.14.14.41:443 fail
Nov 05 06:25:07 vps1 nginx[1527]: nginx: [emerg] bind() to 141.14.14.41:80 faile
Nov 05 06:25:07 vps1 nginx[1527]: nginx: [emerg] bind() to 141.14.14.41:443 fail
Nov 05 06:25:07 vps1 nginx[1527]: nginx: [emerg] still could not bind()
Nov 05 06:25:07 vps1 systemd[1]: nginx.service: Control process exited, code=exi
Nov 05 06:25:07 vps1 systemd[1]: Failed to start A high performance web server a
Nov 05 06:25:07 vps1 systemd[1]: nginx.service: Unit entered failed state.
Nov 05 06:25:07 vps1 systemd[1]: nginx.service: Failed with result 'exit-code'.
~
~
~
~
~
~
lines 1-18/18 (END)...skipping...
● nginx.service - A high performance web server and a reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Sun 2017-11-05 06:25:07 WET; 1h 50min ago
  Process: 5062 ExecStop=/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 --pidfile /run/nginx.pid (code=exited, status=1/FAILURE)
  Process: 1527 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=1/FAILURE)
  Process: 1522 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
 Main PID: 1486 (code=exited, status=0/SUCCESS)

Nov 05 06:25:06 vps1 nginx[1527]: nginx: [emerg] bind() to 141.14.14.41:443 failed (98: Address already in use)
Nov 05 06:25:06 vps1 nginx[1527]: nginx: [emerg] bind() to 141.14.14.41:80 failed (98: Address already in use)
Nov 05 06:25:06 vps1 nginx[1527]: nginx: [emerg] bind() to 141.14.14.41:443 failed (98: Address already in use)
Nov 05 06:25:07 vps1 nginx[1527]: nginx: [emerg] bind() to 141.14.14.41:80 failed (98: Address already in use)
Nov 05 06:25:07 vps1 nginx[1527]: nginx: [emerg] bind() to 141.14.14.41:443 failed (98: Address already in use)
Nov 05 06:25:07 vps1 nginx[1527]: nginx: [emerg] still could not bind()
Nov 05 06:25:07 vps1 systemd[1]: nginx.service: Control process exited, code=exited status=1
Nov 05 06:25:07 vps1 systemd[1]: Failed to start A high performance web server and a reverse proxy server.
Nov 05 06:25:07 vps1 systemd[1]: nginx.service: Unit entered failed state.
Nov 05 06:25:07 vps1 systemd[1]: nginx.service: Failed with result 'exit-code'.

my /etc/nginx/nginx.conf:

user www-data;
worker_processes auto;
pid /run/nginx.pid;

events {
    worker_connections 1024;
    # multi_accept on;
}

http {

    ##
    # Basic Settings
    ##

    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 65;
    types_hash_max_size 2048;
    # server_tokens off;

    # server_names_hash_bucket_size 64;
    # server_name_in_redirect off;

    include /etc/nginx/mime.types;
    default_type application/octet-stream;

    ##
    # SSL Settings
    ##

    ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
    ssl_prefer_server_ciphers on;

    ##
    # Logging Settings
    ##

    access_log /var/log/nginx/access.log;
    error_log /var/log/nginx/error.log;

    ##
    # Gzip Settings
    ##

    gzip on;
    gzip_disable "msie6";

    gzip_vary on;
    gzip_proxied any;
    gzip_comp_level 6;
    gzip_buffers 16 8k;
    gzip_http_version 1.1;
    gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

    ##
    # Virtual Host Configs
    ##

    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;
    server_names_hash_bucket_size 128;
client_max_body_size 100M;
client_body_buffer_size 16k;

fastcgi_buffers 8 16k;
fastcgi_buffer_size 32k;
fastcgi_connect_timeout 180;
fastcgi_send_timeout 180;
fastcgi_read_timeout 180;
}


#mail {
#   # See sample authentication script at:
#   # http://wiki.nginx.org/ImapAuthenticateWithApachePhpScript
# 
#   # auth_http localhost/auth.php;
#   # pop3_capabilities "TOP" "USER";
#   # imap_capabilities "IMAP4rev1" "UIDPLUS";
# 
#   server {
#       listen     localhost:110;
#       protocol   pop3;
#       proxy      on;
#   }
# 
#   server {
#       listen     localhost:143;
#       protocol   imap;
#       proxy      on;
#   }
#}

Nginx Log:

2017/11/05 06:25:08 [emerg] 1703#1703: bind() to 141.14.14.41:80 failed (98: Address already in use)
2017/11/05 06:25:08 [emerg] 1703#1703: bind() to 141.14.14.41:443 failed (98: Address already in use)
2017/11/05 06:25:08 [emerg] 1703#1703: bind() to 141.14.14.41:80 failed (98: Address already in use)
2017/11/05 06:25:08 [emerg] 1703#1703: bind() to 141.14.14.41:443 failed (98: Address already in use)
2017/11/05 06:25:08 [emerg] 1703#1703: bind() to 141.14.14.41:80 failed (98: Address already in use)
2017/11/05 06:25:08 [emerg] 1703#1703: bind() to 141.14.14.41:443 failed (98: Address already in use)
2017/11/05 06:25:08 [emerg] 1703#1703: bind() to 141.14.14.41:80 failed (98: Address already in use)
2017/11/05 06:25:08 [emerg] 1703#1703: bind() to 141.14.14.41:443 failed (98: Address already in use)
2017/11/05 06:25:08 [emerg] 1703#1703: bind() to 141.14.14.41:80 failed (98: Address already in use)
2017/11/05 06:25:08 [emerg] 1703#1703: bind() to 141.14.14.41:443 failed (98: Address already in use)
2017/11/05 06:25:08 [emerg] 1703#1703: still could not bind()
2017/11/05 06:27:39 [emerg] 1978#1978: bind() to 141.14.14.41:80 failed (98: Address already in use)
2017/11/05 06:27:39 [emerg] 1978#1978: bind() to 141.14.14.41:443 failed (98: Address already in use)
2017/11/05 06:27:39 [emerg] 1978#1978: bind() to 141.14.14.41:80 failed (98: Address already in use)
2017/11/05 06:27:39 [emerg] 1978#1978: bind() to 141.14.14.41:443 failed (98: Address already in use)
2017/11/05 06:27:39 [emerg] 1978#1978: bind() to 141.14.14.41:80 failed (98: Address already in use)
2017/11/05 06:27:39 [emerg] 1978#1978: bind() to 141.14.14.41:443 failed (98: Address already in use)
2017/11/05 06:27:39 [emerg] 1978#1978: bind() to 141.14.14.41:80 failed (98: Address already in use)
2017/11/05 06:27:39 [emerg] 1978#1978: bind() to 141.14.14.41:443 failed (98: Address already in use)
2017/11/05 06:27:39 [emerg] 1978#1978: bind() to 141.14.14.41:80 failed (98: Address already in use)
2017/11/05 06:27:39 [emerg] 1978#1978: bind() to 141.14.14.41:443 failed (98: Address already in use)
2017/11/05 06:27:39 [emerg] 1978#1978: still could not bind()
2017/11/05 06:30:10 [emerg] 2076#2076: bind() to 141.14.14.41:80 failed (98: Address already in use)
2017/11/05 06:30:10 [emerg] 2076#2076: bind() to 141.14.14.41:443 failed (98: Address already in use)
2017/11/05 06:30:10 [emerg] 2076#2076: bind() to 141.14.14.41:80 failed (98: Address already in use)
2017/11/05 06:30:10 [emerg] 2076#2076: bind() to 141.14.14.41:443 failed (98: Address already in use)
2017/11/05 06:30:10 [emerg] 2076#2076: bind() to 141.14.14.41:80 failed (98: Address already in use)
2017/11/05 06:30:10 [emerg] 2076#2076: bind() to 141.14.14.41:443 failed (98: Address already in use)
2017/11/05 06:30:10 [emerg] 2076#2076: bind() to 141.14.14.41:80 failed (98: Address already in use)
2017/11/05 06:30:10 [emerg] 2076#2076: bind() to 141.14.14.41:443 failed (98: Address already in use)
2017/11/05 06:30:10 [emerg] 2076#2076: bind() to 141.14.14.41:80 failed (98: Address already in use)
2017/11/05 06:30:10 [emerg] 2076#2076: bind() to 141.14.14.41:443 failed (98: Address already in use)
2017/11/05 06:30:10 [emerg] 2076#2076: still could not bind()
2017/11/05 06:32:41 [emerg] 2208#2208: bind() to 141.14.14.41:80 failed (98: Address already in use)
2017/11/05 06:32:41 [emerg] 2208#2208: bind() to 141.14.14.41:443 failed (98: Address already in use)
2017/11/05 06:32:41 [emerg] 2208#2208: bind() to 141.14.14.41:80 failed (98: Address already in use)
2017/11/05 06:32:41 [emerg] 2208#2208: bind() to 141.14.14.41:443 failed (98: Address already in use)
2017/11/05 06:32:41 [emerg] 2208#2208: bind() to 141.14.14.41:80 failed (98: Address already in use)

Ubuntu 16.04-64b / nginx 1.10.3 /php7.0-fpm

I trying just see status: service nginx status:

nginx.service - A high performance web server and a reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Sun 2017-11-05 09:00:51 WET; 3h 5min ago
  Process: 5062 ExecStop=/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 --pidfile /run/nginx.pid (code=exited, status=1/FAILURE)
  Process: 8636 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=1/FAILURE)
  Process: 8633 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
 Main PID: 1486 (code=exited, status=0/SUCCESS)

Nov 05 09:00:50 vps1 nginx[8636]: nginx: [emerg] bind() to 141.14.14.41:443 failed (98: Address already in use)
Nov 05 09:00:50 vps1 nginx[8636]: nginx: [emerg] bind() to 141.14.14.41:80 failed (98: Address already in use)
Nov 05 09:00:50 vps1 nginx[8636]: nginx: [emerg] bind() to 141.14.14.41:443 failed (98: Address already in use)
Nov 05 09:00:51 vps1 nginx[8636]: nginx: [emerg] bind() to 141.14.14.41:80 failed (98: Address already in use)
Nov 05 09:00:51 vps1 nginx[8636]: nginx: [emerg] bind() to 141.14.14.41:443 failed (98: Address already in use)
Nov 05 09:00:51 vps1 nginx[8636]: nginx: [emerg] still could not bind()
Nov 05 09:00:51 vps1 systemd[1]: nginx.service: Control process exited, code=exited status=1
Nov 05 09:00:51 vps1 systemd[1]: Failed to start A high performance web server and a reverse proxy server.
Nov 05 09:00:51 vps1 systemd[1]: nginx.service: Unit entered failed state.
Nov 05 09:00:51 vps1 systemd[1]: nginx.service: Failed with result 'exit-code'.

If I restart the machine in force, reboot now (I just do this again), the nginx pid reappears(/run/nginx.pid), and monit returns to monitor the process:

https://imgur.com/a/IWfch

but of course no one may have to restart vps just to do a nginx status or nginx restart because there is no more nginx pid on linux server!

I really appreciate your help.

  • Another process is blocking ports 80 and 443. When the problem happens, enter netstat -tulpan | egrep ':80|:443' to find the processes that are blocking the ports. Depending on the outcome, dig further to troubleshoot. – Thomas Nov 5 '17 at 11:40
  • Thanks @Thomas.. my output: pastebin.com/1VhyQxfT – Ariane Martins Gomes Do Rego Nov 5 '17 at 11:46
  • There is running a nginx already. Did you try to restart nginx. Is the problem rather related to monit? – Thomas Nov 5 '17 at 11:54
  • No, the monit is ok, it just can not work with nginx, because the problem is the nginx pid disappearing from linux server !!, I have no control over nginx, I can not see status, restart etc .. I have to reboot the machine as I say in question :( sudo find /run -iname nginx*.pid without pid, I can't nothing .. I add in question output about service nginx status :( – Ariane Martins Gomes Do Rego Nov 5 '17 at 12:05
  • Hm....but the nginx server is running as you can see in the output of netstat -tulpan ... with the two PIDs 5080 and 5081. What do you plan to do with the PID? Normally on Ubuntu 16.04 */run/nginx.pid` does hold the process of the master process. You can also check the PIDs with ps aux | grep nginx. – Thomas Nov 5 '17 at 12:20
0

First try: pkill nginx && service nginx restart

If the file still gets deleted, then you can use ftrace to find which process is deleting your file:

trace-cmd record -e \*unlink\*

Wait for the file to be deleted, press CTRL+C to stop trace-cmd record ..., then run:

trace-cmd report

See also: https://serverfault.com/a/310477/30946

Also make sure that pid in /usr/lib/systemd/system/nginx.service or /etc/systemd/system/nginx.service files is pointing to the same file as in /etc/nginx/nginx.conf

  • sorry for long delay. I always use: pkill nginx OR fuser -k 443/tcp && service nginx START to retrieve the PID nginx link back to /run/nginx/, do not even need to restart nginx, just start! The PID file link back to system, but disappears again from the system between 24h and 72h! I've already followed all its guidelines except the trace-cmd part, I'll do it and I'll come back to say, this will re-trace the process that are you deleting my link every day? or will it solve the problem so it will not be deleted? I've been even thinking about making it immutable, chattr + i Thankyou s.m. – Ariane Martins Gomes Do Rego Nov 27 '17 at 22:12
  • trace-cmd is to find what is deleting your file – Mircea Vutcovici Nov 28 '17 at 3:13
  • @Vutcovici Thanks a lot for your help, my main objective is to find out, even to be useful for future experience, two servers stopped with the disappearance of the PID file, a VPS still has the problem, so I was able to do what you taught:pastebin.com/ZDj1AV9M I'm taking a long time to come here to answer, because I keep monitoring the VPS to see the changes and reactions, waiting for reactions to the attempts that I make. I really want to figure this out and why the problem occurs! This is important for other future cases. – Ariane Martins Gomes Do Rego Dec 4 '17 at 11:39
  • It seems that only fail2ban-server deleted files. You can try to disable it for a few hours to see if this is indeed the root cause. If fail2ban-server is the problem, I would trace it using strace -e file fail2ban....... – Mircea Vutcovici Dec 5 '17 at 17:48
  • I think I already found the problem, I created two server blocks in my vhosts, with http and another one for https. and I had not commented the line list https in http block and the line list http in https block, I will be sure and to create a good explanation and to play in pastebin for you to see, I will take a week to do this, because I am doing tests. Thanks @Miscea – Ariane Martins Gomes Do Rego Dec 14 '17 at 19:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.