nginx routing broken after server reboot

Question

On a CentOS server, I have an nginx installation that acts as a proxy to a couple web applications. Nginx is configured to route requests to different hostnames to different backend servers.

This server has been setup a few months ago and everything worked properly until now. After a reboot (this was probably also the first reboot after nginx was configured), nginx does no longer route properly: it routes all request to the same backend, regardless of the hostname!

The following has been verified:

• All backend applications are running and listening to their respective ports (verified with netstat)
• The nginx configuration has no errors (verified with sudo nginx -t)
• Nginx is reading all configuration files properly (verified with sudo nginx -T)
• Nginx is not outputting any errors (/var/log/nginx/error.log is empty, sudo journalctl -u nginx is not displaying anything that indicates an error)

The server in question contains 4 applications (2 custom backend applications named "organizer" and "integration" in the configuration files, a mattermost server and a gitea GIT server). All http traffic is redirected to https.

Despite the configuration, everything redirects to the organizer. If the organizer backend is closed, nothing works.

The following is the configuration (domains edited out) as output by nginx, irrelevant parts have been removed:

# configuration file /etc/nginx/nginx.conf:

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

# Load dynamic modules. See /usr/share/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 1024;
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    # Load modular configuration files from the /etc/nginx/conf.d directory.
    # See http://nginx.org/en/docs/ngx_core_module.html#include
    # for more information.
    include /etc/nginx/conf.d/*.conf;
}

# configuration file /usr/share/nginx/modules/mod-http-geoip.conf:
load_module "/usr/lib64/nginx/modules/ngx_http_geoip_module.so";

# configuration file /usr/share/nginx/modules/mod-http-image-filter.conf:
load_module "/usr/lib64/nginx/modules/ngx_http_image_filter_module.so";

# configuration file /usr/share/nginx/modules/mod-http-perl.conf:
load_module "/usr/lib64/nginx/modules/ngx_http_perl_module.so";

# configuration file /usr/share/nginx/modules/mod-http-xslt-filter.conf:
load_module "/usr/lib64/nginx/modules/ngx_http_xslt_filter_module.so";

# configuration file /usr/share/nginx/modules/mod-mail.conf:
load_module "/usr/lib64/nginx/modules/ngx_mail_module.so";

# configuration file /usr/share/nginx/modules/mod-stream.conf:
load_module "/usr/lib64/nginx/modules/ngx_stream_module.so";

# configuration file /etc/nginx/conf.d/http_to_https.conf:
server {
    listen 80 default_server;
    listen [::]:80 default_server;
    server_name _;
    return 301 https://$host$request_uri;
}

# configuration file /etc/nginx/conf.d/integration.conf:
server {
    listen          443 ssl http2;
    listen          [::]:443 ssl http2;
    server_name     integration.mydomain.org;

    ssl on;
    ssl_certificate     /etc/nginx/ssl/mydomain_wildcard.pem;
    ssl_certificate_key /etc/nginx/ssl/mydomain_wildcard.key;

    location / {
        proxy_pass      http://localhost:3080;
        proxy_set_header    Host        $host;
        proxy_set_header    X-Real-IP   $remote_addr;
        proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header    X-Client-Verify SUCCESS;
        proxy_set_header    X-Client-DN $ssl_client_s_dn;
        proxy_set_header    X-SSL-Subject   $ssl_client_s_dn;
        proxy_set_header    X-SSL-Issuer    $ssl_client_i_dn;
        proxy_read_timeout  1800;
        proxy_connect_timeout   1800;
    }
}

# configuration file /etc/nginx/conf.d/mattermost.conf:
server {
    listen          443 ssl http2;
    listen          [::]:443 ssl http2;
    server_name     chat.mydomain.org;

    ssl on;
    ssl_certificate     /etc/nginx/ssl/mydomain_wildcard.pem;
    ssl_certificate_key /etc/nginx/ssl/mydomain_wildcard.key;

    location / {
        proxy_pass      http://localhost:8065;
        proxy_set_header    Host        $host;
        proxy_set_header    X-Real-IP   $remote_addr;
        proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header    X-Client-Verify SUCCESS;
        proxy_set_header    X-Client-DN $ssl_client_s_dn;
        proxy_set_header    X-SSL-Subject   $ssl_client_s_dn;
        proxy_set_header    X-SSL-Issuer    $ssl_client_i_dn;
        proxy_read_timeout  1800;
        proxy_connect_timeout   1800;
    }
}

# configuration file /etc/nginx/conf.d/git.conf:
server {
    listen          443 ssl http2;
    listen          [::]:443 ssl http2;
    server_name     git.mydomain.org;

    ssl on;
    ssl_certificate /etc/letsencrypt/live/git.mydomain.org/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/git.mydomain.org/privkey.pem; # managed by Certbot

    location / {
        proxy_pass      http://localhost:3000;
        proxy_set_header    Host        $host;
        proxy_set_header    X-Real-IP   $remote_addr;
        proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header    X-Client-Verify SUCCESS;
        proxy_set_header    X-Client-DN $ssl_client_s_dn;
        proxy_set_header    X-SSL-Subject   $ssl_client_s_dn;
        proxy_set_header    X-SSL-Issuer    $ssl_client_i_dn;
        proxy_read_timeout  1800;
        proxy_connect_timeout   1800;
    }
}

# configuration file /etc/nginx/conf.d/organizer.conf:
server {
    listen          443 ssl http2;
    listen          [::]:443 ssl http2;
    server_name     organizer.mydomain.org;

    ssl on;
    ssl_certificate     /etc/nginx/ssl/mydomain_wildcard.pem;
    ssl_certificate_key /etc/nginx/ssl/mydomain_wildcard.key;

    location / {
        proxy_pass      http://localhost:3080;
        proxy_set_header    Host        $host;
        proxy_set_header    X-Real-IP   $remote_addr;
        proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header    X-Client-Verify SUCCESS;
        proxy_set_header    X-Client-DN $ssl_client_s_dn;
        proxy_set_header    X-SSL-Subject   $ssl_client_s_dn;
        proxy_set_header    X-SSL-Issuer    $ssl_client_i_dn;
        proxy_read_timeout  1800;
        proxy_connect_timeout   1800;
    }
}

This issue is quite puzzling, and I couldn't find anything wrong except the server not working as intended. Does anyone see (or know of...) a reason why nginx would ignore its configuration and route everything to the same backend?

Additional information:

• Nginx is on version 1.12.2 (and hasn't been updated since the server was installed)

Answer 1

Can you provide us with the version of the Nginx you are running now? Any difference with the version you installed?

Also, have your tried to modify the config to run for one domain at a time in order to find more abnormal behavior and collect more info about this issue?

Answer 2

The cause of the issue has been found, and had nothing to do with nginx in the first place!

Before I was using nginx, at the time there was only a single backend application. I had some firewall rules setup to forward ports 80 and 443 to the single backend. When nginx was installed, the firewall rules were removed to make space for nginx - but apparently, not permanently!

Hence, everything worked back then. Until reboot, when the firewall rules are reloaded to their previous configuration. Despite checking netstat and iptables, everything looked fine.

Found the issue when trying nikpelgr's suggestion to try a single application at a time. I discovered that it all didn't matter. After shutting down nginx, just to try, my server was still up...

Lessons learned:

• Try a reboot of your server, after everything is setup. If it breaks, at least everything will be fresh in mind...
• Netstat doesn't take into account firewalld.