I have a fairly standard HAProxy / Web Cluster setup, which is running perfectly fine - as long as I don't enable the Shorewall firewall on the web cluster servers.
As soon as I do, error messages appear in the HAProxy server's syslog, and the websites served via HAProxy returns 503.
Take the HAProxy server and one of the web servers as an example.
The HAProxy config looks like this:
global
daemon
maxconn 8192
log 127.0.0.1 local5 info
defaults
mode http
timeout connect 5000ms
timeout client 50000ms
timeout server 50000ms
log global
option httplog
option dontlognull
option http-server-close
option tcplog clf
option httplog clf
stats enable
stats refresh 10s
stats uri /haprsts
listen http-in
bind *:80
mode http
option forwardfor
default_backend www_servers
backend www_servers
balance leastconn
cookie SERVERID insert indirect nocache
server server1 XXX.XXX.XXX.XXX:9009 maxconn 128 check cookie server1
server server2 YYY.YYY.YYY.YYY:9009 maxconn 128 check cookie server2
listen https-in
mode http
bind *:443 ssl crt /etc/haproxy/certs/cert.pem
option forwardfor
reqadd X-Forwarded-Proto:\ https
default_backend www_servers_ssl
backend www_servers_ssl
balance leastconn
cookie SERVERID insert indirect nocache
server server1 XXX.XXX.XXX.XXX:9009 maxconn 128 check inter 3000 fall 2 rise 2 cookie server1
server server2 YYY.YYY.YYY.YYY:9009 maxconn 128 check inter 3000 fall 2 rise 2 cookie server2
XXX.XXX.XXX.XXX and YYY.YYY.YYY.YYY are public IP addresses.
Apache and the vhosts configs on the web server, are configured to listen to port 9009, as per the HAProxy config. Everything is running fine up to this point.
Now, I switch on Shorewall Firewall on the web server, with a firewall rules file that looks like this:
#SECTION ALL
#SECTION ESTABLISHED
#SECTION RELATED
SECTION NEW
ACCEPT net:ZZZ.ZZZ.ZZZ.ZZZ $FW:XXX.XXX.XXX.XXX TCP 9009
ACCEPT net:WWW.WWW.WWW.WWW $FW:XXX.XXX.XXX.XXX TCP 9009
# Drop Ping from the "bad" net zone.. and prevent your log from being flooded..
Ping(DROP) net $FW
# Permit all ICMP traffic FROM the firewall TO the net zone
ACCEPT $FW net icmp
ZZZ.ZZZ.ZZZ.ZZZ is the primary HAProxy server, and WWW.WWW.WWW.WWW is the secondary HAProxy server (and not relevant right now to this problem).
And all web traffic dies when the firewall is enabled on the web server.
The HAProxy server starts sending files like these to the syslog: Message from syslogd@localhost at Sep 30 12:59:38 ... haproxy[14631]: backend www_servers has no server available!
And the web server returns 503 errors.
There are other rules in the rules file, which for instance allows SSH on a certain port from a specific IP address, and those rules all work, so I assume (although I could be wrong) it has little to do with the Shorewall firewall rules per say.
Is it so, that HAProxy needs additional ports open on the web server (this is what I suspect) and if so what would those undocumented ports be, or does anyone know what could be wrong here?
Any help would be much appreciated.
Cheers, /j.