0

I'm trying to get an nginx reverse proxy setup to work without success. Nginx runs as a reverse proxy in front of some docker containers on the same host.

I suspect a network configuration issue: from a laptop I can reach the containers in direct, but host to container communication (ie: telnet test) fails on timeout (in Nginx i get a 504 gateway timed out). From a routing perspective, the host is connected to 2 /24 out of which I assign a number of /30 within container's dedicated docker networks.

Here are some config extract:

  1. sample nginx enabled site:

    server { listen 80; listen [::]:80; server_name kibana.example.net;

    location /.well-known/acme-challenge {
        root /var/www/letsencrypt;
        try_files $uri $uri/ =404;
    }
        location / {
        allow 192.168.0.0/16;
        allow ipv6::/62;
        deny all;        proxy_bind 192.168.11.85 transparent ;
        proxy_pass http://log.example.net:5601 ;
    }}
    
  2. nginx error log:

    192.168.12.15 - - [23/Mar/2019:23:17:27 +0100] "GET /favicon.ico HTTP/1.1" 504 176 "https://kibana.example.net/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/11.0 Safari/605.1.15 Epiphany/605.1.15"

  3. ip details:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp5s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 4c:ed:fb:91:28:0f brd ff:ff:ff:ff:ff:ff
3: enp6s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 4c:ed:fb:91:28:10 brd ff:ff:ff:ff:ff:ff
4: enp7s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 4c:ed:fb:91:28:11 brd ff:ff:ff:ff:ff:ff
5: enp8s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 4c:ed:fb:91:28:12 brd ff:ff:ff:ff:ff:ff
6: enp6s0.10@enp6s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-untrusted state UP group default qlen 1000
    link/ether 4c:ed:fb:91:28:10 brd ff:ff:ff:ff:ff:ff
7: enp6s0.11@enp6s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-trusted state UP group default qlen 1000
    link/ether 4c:ed:fb:91:28:10 brd ff:ff:ff:ff:ff:ff
8: br-trusted: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 4c:ed:fb:91:28:10 brd ff:ff:ff:ff:ff:ff
    inet 192.168.11.6/24 brd 192.168.11.255 scope global noprefixroute br-trusted
       valid_lft forever preferred_lft forever
    inet6 ipv6_prefix_1:df75:7ad6:ec88:9c1e/64 scope global dynamic noprefixroute 
       valid_lft 86390sec preferred_lft 14390sec
    inet6 fe80::e84e:85e7:87ba:22ca/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
9: br-untrusted: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 4c:ed:fb:91:28:10 brd ff:ff:ff:ff:ff:ff
    inet 192.168.10.6/24 brd 192.168.10.255 scope global noprefixroute br-untrusted
       valid_lft forever preferred_lft forever
    inet6 ipv6_prefix_0:188f:a7f2:3b03:ed8a/64 scope global dynamic noprefixroute 
       valid_lft 86391sec preferred_lft 14391sec
    inet6 fe80::c3cd:67f1:afc8:44a7/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
10: br-dfcb991bef15: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:63:fa:e0:c3 brd ff:ff:ff:ff:ff:ff
    inet 192.168.11.85/30 brd 192.168.11.87 scope global br-dfcb991bef15
       valid_lft forever preferred_lft forever
    inet6 ipv6_prefix_1:aaaa:aaaa:6:1/118 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::42:63ff:fefa:e0c3/64 scope link 
       valid_lft forever preferred_lft forever
    inet6 fe80::1/64 scope link 
       valid_lft forever preferred_lft forever
11: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:6c:b2:d7:b2 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 ipv6_prefix_1:aaaa:aaaa:ff00:1/104 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::42:6cff:feb2:d7b2/64 scope link 
       valid_lft forever preferred_lft forever
    inet6 fe80::1/64 scope link 
       valid_lft forever preferred_lft forever
12: br-56dc3ad66311: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:7d:97:43:e1 brd ff:ff:ff:ff:ff:ff
    inet 192.168.11.65/30 brd 192.168.11.67 scope global br-56dc3ad66311
       valid_lft forever preferred_lft forever
    inet6 ipv6_prefix_1:aaaa:aaaa:1:1/118 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::42:7dff:fe97:43e1/64 scope link 
       valid_lft forever preferred_lft forever
    inet6 fe80::1/64 scope link 
       valid_lft forever preferred_lft forever
13: br-85e3079fdc7f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:bc:a2:93:67 brd ff:ff:ff:ff:ff:ff
    inet 192.168.11.81/30 brd 192.168.11.83 scope global br-85e3079fdc7f
       valid_lft forever preferred_lft forever
    inet6 ipv6_prefix_1:aaaa:aaaa:5:1/118 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::42:bcff:fea2:9367/64 scope link 
       valid_lft forever preferred_lft forever
    inet6 fe80::1/64 scope link 
       valid_lft forever preferred_lft forever
14: br-c5fd7e123cbe: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:c1:9d:71:c9 brd ff:ff:ff:ff:ff:ff
    inet 192.168.11.89/30 brd 192.168.11.91 scope global br-c5fd7e123cbe
       valid_lft forever preferred_lft forever
    inet6 ipv6_prefix_1:aaaa:aaaa:7:1/118 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::42:c1ff:fe9d:71c9/64 scope link 
       valid_lft forever preferred_lft forever
    inet6 fe80::1/64 scope link 
       valid_lft forever preferred_lft forever
15: br-ca322384e201: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:ce:40:6b:e4 brd ff:ff:ff:ff:ff:ff
    inet 192.168.11.73/30 brd 192.168.11.75 scope global br-ca322384e201
       valid_lft forever preferred_lft forever
    inet6 ipv6_prefix_1:aaaa:aaaa:3:1/118 scope global tentative 
       valid_lft forever preferred_lft forever
    inet6 fe80::1/64 scope link tentative 
       valid_lft forever preferred_lft forever
16: br-7bab57b5a88a: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:ae:d0:7c:84 brd ff:ff:ff:ff:ff:ff
    inet 192.168.10.69/30 brd 192.168.10.71 scope global br-7bab57b5a88a
       valid_lft forever preferred_lft forever
    inet6 ipv6_prefix_0:aaaa:aaaa:1:1/118 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::42:aeff:fed0:7c84/64 scope link 
       valid_lft forever preferred_lft forever
    inet6 fe80::1/64 scope link 
       valid_lft forever preferred_lft forever
17: br-95986f962395: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:ec:b3:fd:71 brd ff:ff:ff:ff:ff:ff
    inet 192.168.11.69/30 brd 192.168.11.71 scope global br-95986f962395
       valid_lft forever preferred_lft forever
    inet6 ipv6_prefix_1:aaaa:aaaa:2:1/118 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::42:ecff:feb3:fd71/64 scope link 
       valid_lft forever preferred_lft forever
    inet6 fe80::1/64 scope link 
       valid_lft forever preferred_lft forever
18: br-9ddddcd915bc: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:44:a2:b2:29 brd ff:ff:ff:ff:ff:ff
    inet 192.168.11.77/30 brd 192.168.11.79 scope global br-9ddddcd915bc
       valid_lft forever preferred_lft forever
    inet6 ipv6_prefix_1:aaaa:aaaa:4:1/118 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::42:44ff:fea2:b229/64 scope link 
       valid_lft forever preferred_lft forever
    inet6 fe80::1/64 scope link 
       valid_lft forever preferred_lft forever
19: br-bb962c92121b: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:62:21:f3:41 brd ff:ff:ff:ff:ff:ff
    inet 192.168.10.65/30 brd 192.168.10.67 scope global br-bb962c92121b
       valid_lft forever preferred_lft forever
    inet6 ipv6_prefix_0:aaaa:aaaa:0:1/118 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::42:62ff:fe21:f341/64 scope link 
       valid_lft forever preferred_lft forever
    inet6 fe80::1/64 scope link 
       valid_lft forever preferred_lft forever
20: br-e325b5818bc3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:d3:fb:06:4a brd ff:ff:ff:ff:ff:ff
    inet 192.168.10.73/30 brd 192.168.10.75 scope global br-e325b5818bc3
       valid_lft forever preferred_lft forever
    inet6 ipv6_prefix_0:aaaa:aaaa:2:1/118 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::42:d3ff:fefb:64a/64 scope link 
       valid_lft forever preferred_lft forever
    inet6 fe80::1/64 scope link 
       valid_lft forever preferred_lft forever
22: vethc8686a1@if21: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-dfcb991bef15 state UP group default 
    link/ether ea:b1:66:9f:e7:fe brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::e8b1:66ff:fe9f:e7fe/64 scope link 
       valid_lft forever preferred_lft forever
28: vethfd7ab7c@if27: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-e325b5818bc3 state UP group default 
    link/ether 3a:85:10:98:fd:aa brd ff:ff:ff:ff:ff:ff link-netnsid 7
    inet6 fe80::3885:10ff:fe98:fdaa/64 scope link 
       valid_lft forever preferred_lft forever
30: veth50b6388@if29: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-7bab57b5a88a state UP group default 
    link/ether 0e:d3:0e:3d:28:d6 brd ff:ff:ff:ff:ff:ff link-netnsid 4
    inet6 fe80::cd3:eff:fe3d:28d6/64 scope link 
       valid_lft forever preferred_lft forever
32: veth107b7de@if31: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-9ddddcd915bc state UP group default 
    link/ether be:28:6f:7b:bb:b1 brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet6 fe80::bc28:6fff:fe7b:bbb1/64 scope link 
       valid_lft forever preferred_lft forever
34: veth2c1228b@if33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-56dc3ad66311 state UP group default 
    link/ether 32:1a:1d:b8:1b:92 brd ff:ff:ff:ff:ff:ff link-netnsid 2
    inet6 fe80::301a:1dff:feb8:1b92/64 scope link 
       valid_lft forever preferred_lft forever
36: vethb498b24@if35: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-bb962c92121b state UP group default 
    link/ether ea:24:23:3f:e2:2c brd ff:ff:ff:ff:ff:ff link-netnsid 3
    inet6 fe80::e824:23ff:fe3f:e22c/64 scope link 
       valid_lft forever preferred_lft forever
182: veth7e01097@if181: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-85e3079fdc7f state UP group default 
    link/ether 2a:56:6e:f9:df:11 brd ff:ff:ff:ff:ff:ff link-netnsid 5
    inet6 fe80::2856:6eff:fef9:df11/64 scope link 
       valid_lft forever preferred_lft forever
root@server:/var/log/nginx# 

4. The routing table:
root@server:/var/log/nginx# ip r
default via 192.168.10.1 dev br-untrusted proto static metric 426 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 
192.168.10.0/24 dev br-untrusted proto kernel scope link src 192.168.10.6 metric 426 
192.168.10.64/30 dev br-bb962c92121b proto kernel scope link src 192.168.10.65 
192.168.10.68/30 dev br-7bab57b5a88a proto kernel scope link src 192.168.10.69 
192.168.10.72/30 dev br-e325b5818bc3 proto kernel scope link src 192.168.10.73 
192.168.10.192/26 via 192.168.10.5 dev br-untrusted proto static metric 426 
192.168.11.0/24 dev br-trusted proto kernel scope link src 192.168.11.6 metric 425 
192.168.11.64/30 dev br-56dc3ad66311 proto kernel scope link src 192.168.11.65 
192.168.11.68/30 dev br-95986f962395 proto kernel scope link src 192.168.11.69 linkdown 
192.168.11.72/30 dev br-ca322384e201 proto kernel scope link src 192.168.11.73 linkdown 
192.168.11.76/30 dev br-9ddddcd915bc proto kernel scope link src 192.168.11.77 
192.168.11.80/30 dev br-85e3079fdc7f proto kernel scope link src 192.168.11.81 
192.168.11.84/30 dev br-dfcb991bef15 proto kernel scope link src 192.168.11.85 
192.168.11.88/30 dev br-c5fd7e123cbe proto kernel scope link src 192.168.11.89 linkdown 
192.168.12.0/23 via 192.168.11.1 dev br-trusted proto static metric 425
  1. docker network inspect:
root@server:/var/log/nginx# docker network inspect nw_log
[
    {
        "Name": "nw_log",
        "Id": "dfcb991bef1545566e8932ff511c48cec13acb049a8bb23fd49e6787badd120e",
        "Created": "2019-03-23T21:57:17.791878236+01:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": true,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "192.168.11.84/30",
                    "IPRange": "192.168.11.84/30",
                    "Gateway": "192.168.11.85"
                },
                {
                    "Subnet": "ipv6_prefix_1:AAAA:AAAA:6::/118",
                    "IPRange": "ipv6_prefix_1:AAAA:AAAA:6::/118",
                    "Gateway": "ipv6_prefix_1:AAAA:AAAA:6:1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "cc60315d8464859008ba05d5fd986912f6962257915f2a4b467328f01a50dd5c": {
                "Name": "log",
                "EndpointID": "03b30f289a15c1ea20468e0c19c659ff257e42b9df0bfe1119b74ccf1d022323",
                "MacAddress": "02:42:c0:a8:0b:56",
                "IPv4Address": "192.168.11.86/30",
                "IPv6Address": "ipv6_prefix_1:aaaa:aaaa:6:2/118"
            }
        },
        "Options": {
            "com.docker.network.bridge.enable_icc": "true"
        },
        "Labels": {}
    }
]
  • please disregard this question, the issue is not with the reverse proxy setup but at routing level, created another question which is more accurate here : serverfault.com/questions/961344/… – Nicolas Apr 3 at 17:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.