1

I'm running instances of Symfony or Drupal websites on two Debian servers, with Nginx listening to 443, Varnish listening to 80 and passing to nginx on listening custom ports 80** for each vhost.

Recently I added a new website to one of the servers. Then I began to run in this quite documented error nginx: [emerg] bind() to [::]:80 failed (98: Address already in use).

Despite there is no nginx server block at all listening to :80 port, neither any server block without listen directive, Nginx began to listen on port 80 all together with the custom ports.

sudo netstat -tlpn| grep nginx
tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      4191/nginx: master  
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      4191/nginx: master  
tcp        0      0 0.0.0.0:8081            0.0.0.0:*               LISTEN      4191/nginx: master  
tcp        0      0 x.x.x.x:8082            0.0.0.0:*               LISTEN      4191/nginx: master  
tcp        0      0 y.y.y.y:8083            0.0.0.0:*               LISTEN      4191/nginx: master  
tcp        0      0 z.z.z.z:8084            0.0.0.0:*               LISTEN      4191/nginx: master  
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      4191/nginx: master  
tcp        0      0 0.0.0.0:8000            0.0.0.0:*               LISTEN      4191/nginx: master  
tcp6       0      0 :::8080                 :::*                    LISTEN      4191/nginx: master  
tcp6       0      0 :::80                   :::*                    LISTEN      4191/nginx: master  
tcp6       0      0 :::8081                 :::*                    LISTEN      4191/nginx: master  
tcp6       0      0 :::443                  :::*                    LISTEN      4191/nginx: master  
tcp6       0      0 :::8000                 :::*                    LISTEN      4191/nginx: master

I already read a bunch of questions and posts about handling dual-stack IPv4 and IPv6 correct new syntax, and tried , AFAIK, all possible syntaxes such as below, no way.

Working directive before crash : listen x.x.x.x:8082; Tried adding listen [::]:8082 ipv6only=on; . No change.

I listed, and killed process many times with sudo fuser -k 80/tcp before restarting systemctl varnish, nginx, even daemon-reload...

Last, I checked my history but can't find what could have caused this sudden behavior. The lone point I'm not sure about is I changed a couple of sysctl.conf params, but hopefully reverted them, just in case, I'm not used to this part od administration : cat /etc/sysctl.conf | grep net.ipv4.conf

#net.ipv4.conf.default.rp_filter=1
#net.ipv4.conf.all.rp_filter=1
#net.ipv4.conf.all.accept_redirects = 0
# net.ipv4.conf.all.secure_redirects = 1
#net.ipv4.conf.all.send_redirects = 0
#net.ipv4.conf.all.accept_source_route = 0
#net.ipv4.conf.all.log_martians = 1

Here's my configuration.

cat /etc/nginx/nginx.conf (relevant 2 lines, no server block in it)

include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;

cat /etc/nginx/conf.d/default.conf

server {
        listen 8000 default_server;
        listen [::]:8000 ipv6only=on default_server;
        server_name _;

        listen 443 ssl default_server;
        listen [::]:443 ssl ipv6only=on default_server;
}

One of the sites-available vhosts (they all follow exactly same pattern) :

server { # this block only redirects www to non www
        listen x.x.x.x:443 ssl;
        server_name www.example.com;

        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
        ssl_certificate /var/www/clients/client0/web3/ssl/example.com-le.crt;
        ssl_certificate_key /var/www/clients/client0/web3/ssl/example.com-le.key;

        return 301 https://example.com$request_uri;
        }

server {
        listen x.x.x.x:443 ssl;
        server_name example.com

        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
        ssl_certificate /var/www/clients/client0/web3/ssl/example.com-le.crt;
        ssl_certificate_key /var/www/clients/client0/web3/ssl/example.com-le.key;

        location / {
            # Pass the request on to Varnish.
            proxy_pass  http://127.0.0.1;
 
            # Pass some headers to the downstream server, so it can identify the host.
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 
            # Tell any web apps like Drupal that the session is HTTPS.
            proxy_set_header X-Forwarded-Proto https;
            proxy_redirect     off;
        }
        
}
server {
        listen x.x.x.x:8082;
#       listen [::]:8082 ipv6only=on;

        server_name example.com www.example.com;

        root   /var/www/example.com/web/public;

        location / {
            # try to serve file directly, fallback to index.php
            try_files $uri /index.php$is_args$args;
        }

       location ~ ^/index\.php(/|$) {
            fastcgi_pass 127.0.0.1:8998;
            fastcgi_split_path_info ^(.+\.php)(/.*)$;
            include fastcgi_params;
            fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name;
            fastcgi_param DOCUMENT_ROOT $realpath_root;
            internal;
        }
        location ~ \.php$ {
           # return 404;
        }

        error_log /var/log/ispconfig/httpd/example.com/error.log;
        access_log /var/log/ispconfig/httpd/example.com/access.log combined;

        location ~ /\. {
                        deny all;
        }

        location ^~ /.well-known/acme-challenge/ {
             access_log off;
             log_not_found off;
             root /usr/local/ispconfig/interface/acme/;
             autoindex off;
             try_files $uri $uri/ =404;
        }

        location = /favicon.ico {
            log_not_found off;
            access_log off;
            expires max;
            add_header Cache-Control "public, must-revalidate, proxy-revalidate";
        }

        location = /robots.txt {
            allow all;
            log_not_found off;
            access_log off;
        }
}

cat /etc/default/varnish relevant part

DAEMON_OPTS="-a :80 \
             -T localhost:6082 \
             -f /etc/varnish/default.vcl \
             -S /etc/varnish/secret \
             -s malloc,3G"

I'm wondering what could have caused a config I'm working with since years to bug ?

I carefully studied these Q&A and a bunch of doc or posts, with no success : Nginx tries to run on port 80 but the configs have been removed ; Nginx will not start (Address already in use) ; nginx - bind() to 0.0.0.0:80 failed (98: Address already in use)

EDIT

Here's the output of nginx -T. (Since body is limited to 30000 characters I had to paste it in pastebin).

  • Please post the output of nginx -T. – Michael Hampton Jun 27 at 16:00
  • @MichaelHampton Proceeding, thanks, just give me a couple of min please, since I need to remove all the real IP and domains – Kojo Jun 27 at 16:03
  • We prefer that posts not be obfuscated when possible. See here for discussion and guidance. – Michael Hampton Jun 27 at 16:18
  • 1
    What's this include /etc/letsencrypt/le_http_01_cert_challenge.conf? Leftover from some old certbot run? It probably shouldn't be there. – Michael Hampton Jun 27 at 16:39
  • 1
    certbot does a challenge response when you get a Let's Encrypt certificate. – Michael Hampton Jun 27 at 16:56
1

Well, thanks to @MichaelHampton, I realized the searched listen directive was hidden in a certbot challenge, called in nginx.conf with an include :

# configuration file /etc/letsencrypt/le_http_01_cert_challenge.conf:
server{listen 80;listen [::]:80;server_name example.org;root /var/lib/letsencrypt/http_01_nonexistent;location = /.well-known/acme-challenge/PlsQNg7nOVxIe6CwwGpco
KTbSudji44JNZVQA57EyNE{default_type text/plain;return 200 PlsQNg7nOVxIe6CwwGpcoKTbSudji44JNZVQA57EyNE.7nkyfxInEw24UW4P7xfgJQGTMXYGQH_mzIOz6F0641Y;}} 

This underlined 2 basic lessons (at least for myself) :

  1. In case of emergency, don't search faster, but slower, taking time to really evaluate each line you read : this include was the first line of my HTTP block !!!
  2. Specifically, for Nginx, that simple CLI Nginx -T is a killer tool : outputting an inline version of every single line of each config file, it gives a powerful way to find immediately the culprit : nginx -T | grep ':80' would have put me on the right track in seconds !
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.