-1

How to configure NGINX to route external request to my stage and production docker hosts

I have 2 FQDNs stage.external.example.net and external.example.net that resolve to the same external public IP address, e.g. 140.240.40.111 (configured on the external DNS)

The HTTPS requests to this IP (e.g. GET https://stage.external.example.net) are then routed to my NGINX server that has one leg on the DMZ e.g. 172.20.180.111 and another to my internal network e.g. 10.222.20.1/16 on which I have 2 docker hosts that resolve internally as follow:

  • stage.internal.example.net => 10.222.20.14
  • internal.example.net => 10.222.20.15

I am trying to configure my NGINX to route:

  • external request stage.external.example.net:443 to internal stage.internal.example.net:443
  • external request external.example.net:443 to internal internal.example.net:443
stage.external.example.net:443 -> 140.240.40.111:443 -> 172.20.180.111:443 (NGINX) -> 10.222.20.14:443 (stage.internal.example.net)

external.example.net:443 -> 140.240.40.111:443 -> 172.20.180.111:443 (NGINX) -> 10.222.20.15:443 (internal.example.net)

I can see that requests are hitting my NGINX in the access.log, but then the request do not seem to be routed forward to my docker hosts.

Hereafter the configuration that I have tried, any pointers are most welcome:

nginx.conf

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

include /usr/share/nginx/modules/*.conf;

events {
  worker_connections 1024;
}

http {
  log_format main '$remote_addr - $remote_user [$time_local] '
                  '"$request" $status $body_bytes_sent '
                  '"$http_referer" "$http_user_agent"';

  access_log /var/log/nginx/access.log main;
  error_log /var/log/nginx/error.log warn; 

  sendfile            on;
  tcp_nopush          on;
  tcp_nodelay         on;
  keepalive_timeout   65;
  types_hash_max_size 2048;

  include             /etc/nginx/mime.types;
  default_type        application/octet-stream;

  include /etc/nginx/conf.d/*.conf;

  upstream internal_stage {
    server 10.222.20.14:443; # docker host stage.internal.example.net
  }

  upstream internal_production {
    server 10.222.20.15:443; # docker host internal.example.net 
  }

  # Forward all requests to stage.external.example.net:443
  server {
    listen 443;
    server_name stage.external.example.net;
    location / {
      proxy_pass http://internal_stage;
    }
  }

 
  # Forward all requests to external.example.net:443
  server {
    listen 443;
    server_name external.example.net;
    location / {
      proxy_pass http://internal_production;
    }
  }
}

UPDATE: Change error_log to info

I can now observe that I am getting a 400 Bad Request Client sent invalid request while reading client request line

1
  • 1
    instead of voting down a legit question, please ask to clarify, so I can progress and learn.
    – zabumba
    May 6, 2021 at 20:59

3 Answers 3

2

seems the backend is https so the proxy_pass should be https also like bellow:

location /upstream {
    proxy_pass https://backend.example.com;
}

Please check securing-http-traffic-upstream

1
  • I had tried with https, also I am defining 443 in the upstream, so I figured that would be enough. following location should be the location, I may try that blindly, but I am not confident that it is how it works. Thanks for trying answer though ;) that's good pointers
    – zabumba
    May 6, 2021 at 20:55
0

RESOLVED by using a stream block and ssl_preread

nginx.conf

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

include /usr/share/nginx/modules/*.conf;

events {
  worker_connections 1024;
}

http {
  log_format main '$remote_addr - $remote_user [$time_local] '
                  '"$request" $status $body_bytes_sent '
                  '"$http_referer" "$http_user_agent"';

  access_log /var/log/nginx/access.service-api.log main;
  error_log /var/log/nginx/error.service-api.log info;

  sendfile            on;
  tcp_nopush          on;
  tcp_nodelay         on;
  keepalive_timeout   65;
  types_hash_max_size 2048;

  include             /etc/nginx/mime.types;
  default_type        application/octet-stream;

  include /etc/nginx/conf.d/*.conf;
}

include /etc/nginx/passthrough.conf;

passthrough.conf

stream {
  log_format basic '$remote_addr [$time_local] '
                 '$protocol $status $bytes_sent $bytes_received '
                 '$session_time "$upstream_addr" '
                 '"$upstream_bytes_sent" "$upstream_bytes_received" "$upstream_connect_time"';

  access_log /var/log/nginx/access.log basic;
  error_log  /var/log/nginx/error.log;

  upstream internal_stage {
    server 10.222.20.14:443 max_fails=5 fail_timeout=300; # docker host stage.internal.domain.net
  }

  upstream internal_production {
    server 10.222.20.15:443 max_fails=5 fail_timeout=300; # docker host stage.internal.domain.net 
  }

  map $ssl_preread_server_name $upstream {
    stage.external.polylabs.net internal_stage;
    external.polylabs.net internal_production;
  }

  server {
    listen 443;
    proxy_pass $upstream;
    ssl_preread on;
    proxy_next_upstream on;        
  }
}
1
  • If you want to terminate TLS in your Docker container nginx, then this is a valid solution. However, this means that you lose the connecting IP address information. Usually TLS is terminated on the outer nginx, and then traffic is passed to upstream services using http protocol. You can also terminate TLS at the outer nginx, and then use https to connect to the nginx running in container. However, that adds extra encryption overhead. May 7, 2021 at 7:04
-1

You seem to be conflating port 443 with HTTPS in your question. If you mean HTTP with encryption on port 443, then that's just https, but hostname:443 tells us nothing about the protocol nor whether there is encryption. Potentially a reason for downvoting the question.

When you use "proxy_pass", nginx forwards the HTTP request to the address specified - but HTTP explicitly supports multiplexing of different (virtual) hostnames on the same IP address. Its worth mentioning that there is a different, supplementary method implemented in TLS for hostname multiplexing (SNI) and here the config you provided conflicts with what I infer as your objective; connecting to https at the backend. Your config is connecting to http. Another reason someone may have downvoted your question. However Nginx's proxy_pass directive will handle the rewriting of the SNI handshake automatically so this is not the cause of your current woes.

Having said that, unless you explicitly tell Nginx not do so, it will validate the certificate presented by the backend server. Does it use a certificate which be validated by nginx? You can test this from the nginx host with openssl's s_client.

the request do not seem to be routed forward to my docker hosts.

How did you come to that conclusion? (omitting the justification for this might be another reason for downvoting your question).

It may be the case - but even if the requests are forwarded to the intended IP address/port, they will result in error if that back end does not think it is the origin server for "stage.external.domain.net".

By far the simplest solution is to tell the backend web server that it really is "stage.external.domain.net". But you can also rewrite the request....

    server_name external.polylabs.net;
    location / {
      proxy_set_header Host "internal.polylabs.net";
      proxy_ssl_verify off;
      proxy_pass https://internal.polylabs.net;
    }
2
  • no need to be condescending while answering in detail WHY people should actually vote my question down. Ain't these communities made to help people? Didn't I make an effort to formulate my question properly? This said, I do appreciate such fully fleshed answer. Hopefully this won't be closed and will be valuable to someone else. Cause that's the point isn't it?
    – zabumba
    May 6, 2021 at 21:57
  • You explicitly asked in the comments above why someone had downvoted your question - I'm just guessing at their reasons. If you take offence when people answer the questions you ask....?
    – symcbean
    May 6, 2021 at 22:00

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .