1

I am running a docker swarm on "swarm.example.com". On the server, there's a container running that can be accessed on "swarm.example.com:3000".

On server "example.com" I'm running an nginx reverse proxy with following rules

server {
    listen 80;
    server_name app.example.com;

    location / {
        proxy_pass http://swarm.example.com:3000;
    }
}

When I try to access app.example.com im getting 502 Bad Gatway error. Am I missing something ?

All servers are running CentOS 7.6

Thanks!

  • Anything in the nginx error log? – Gerald Schneider May 17 at 19:51
  • 1
    also check the selinux audit.log, selinux might cause problems – Gerald Schneider May 17 at 19:53
  • Is your proxy also in a container? If they are both in containers, are the connected to the same network? If the proxy is not in a container and on the same network, then have you 'published' the port 3000? Please consider giving us more details about your containers/networks. – Zoredache May 17 at 21:38
  • @Zoredache no the proxy is on a different physical server – Branislav B. May 17 at 22:23
  • 2
    Don't disable SELinux, fix the problem instead. – Michael Hampton May 18 at 1:46
1

Trying to access the backend via the reverse proxy results in a 502 Bad Gateway error:

$ wget -S --spider http://nginxtest.example.com/
Spider mode enabled. Check if remote file exists.
--2019-05-18 10:12:11--  http://nginxtest.example.com/
Resolving nginxtest.example.com (nginxtest.example.com)... 192.168.15.20
Connecting to nginxtest.example.com (nginxtest.example.com)|192.168.15.20|:80... connected.
HTTP request sent, awaiting response...
  HTTP/1.1 502 Bad Gateway
  Server: nginx/1.12.2
  Date: Sat, 18 May 2019 08:12:11 GMT
  Content-Type: text/html
  Content-Length: 3693
  Connection: keep-alive
  ETag: "5a9e5ebd-e6d"
Remote file does not exist -- broken link!!!

This is most probably because selinux by default doesn't allow outgoing connections for webservers, because that's usually something they don't do.

You will find entries like this in /var/log/nginx/error.log:

2019/05/18 10:12:11 [crit] 1041#0: *5 connect() to 192.168.15.52:3000 failed (13: Permission denied) while connecting to upstream, client: 146.140.37.47, server: _, request: "HEAD / HTTP/1.1", upstream: "http://192.168.15.52:3000/", host: "nginxtest.example.com"

Additionally, you will find entries like this in /var/log/audit/audit.log:

type=AVC msg=audit(1558167131.910:463): avc: denied { name_connect } for pid=1041 comm="nginx" dest=3000 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:ntop_port_t:s0 tclass=tcp_socket permissive=0 type=SYSCALL msg=audit(1558167131.910:463): arch=c000003e syscall=42 success=no exit=-13 a0=8 a1=562671c4eef0 a2=10 a3=7ffcfbc72530 items=0 ppid=1006 pid=1041 auid=4294967295 uid=996 gid=994 euid=996 suid=996 fsuid=996 egid=994 sgid=994 fsgid=994 tty=(none) ses=4294967295 comm="nginx" exe="/usr/sbin/nginx" subj=system_u:system_r:httpd_t:s0 key=(null)

Run the following command to allow nginx to connect to other hosts:

setsebool -P httpd_can_network_connect true

(The parameter -p makes the setting persistent. Otherwise it would be reset after the next reboot.)

And now the proxy works:

$ wget -S --spider http://nginxtest.example.com/
Spider mode enabled. Check if remote file exists.
--2019-05-18 10:15:14--  http://nginxtest.example.com/
Resolving nginxtest.example.com (nginxtest.example.com)... 192.168.15.20
Connecting to nginxtest.example.com (nginxtest.example.com)|192.168.15.20|:80... connected.
HTTP request sent, awaiting response...
  HTTP/1.1 200 OK
  Server: nginx/1.12.2
  Date: Sat, 18 May 2019 08:15:15 GMT
  Content-Type: text/html
  Content-Length: 40
  Connection: keep-alive
  Last-Modified: Sat, 18 May 2019 08:08:16 GMT
  ETag: "5cdfbd70-28"
  Accept-Ranges: bytes
Length: 40 [text/html]
Remote file exists and could contain further links,
but recursion is disabled -- not retrieving.

If you want to know more, there is a very detailed article about nginx and selinux on the nginx website.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.