I have a fairly standard HAProxy / Web Cluster setup, which is running perfectly fine - as long as I don't enable the Shorewall firewall on the web cluster servers.

As soon as I do, error messages appear in the HAProxy server's syslog, and the websites served via HAProxy returns 503.

Take the HAProxy server and one of the web servers as an example.

The HAProxy config looks like this:


global
    daemon
    maxconn 8192
    log 127.0.0.1 local5 info

defaults
    mode http
    timeout connect 5000ms
    timeout client 50000ms
    timeout server 50000ms
    log global
    option httplog
    option dontlognull
    option http-server-close
    option tcplog clf
    option httplog clf
    stats enable
    stats refresh 10s
    stats uri /haprsts

listen http-in
    bind *:80
    mode http
    option forwardfor
    default_backend www_servers

backend www_servers
    balance leastconn
    cookie SERVERID insert indirect nocache
    server server1 XXX.XXX.XXX.XXX:9009 maxconn 128 check cookie server1
    server server2 YYY.YYY.YYY.YYY:9009 maxconn 128 check cookie server2

listen https-in
    mode http
    bind *:443 ssl crt /etc/haproxy/certs/cert.pem
    option forwardfor
    reqadd X-Forwarded-Proto:\ https
    default_backend www_servers_ssl

backend www_servers_ssl
    balance leastconn
    cookie SERVERID insert indirect nocache
    server server1 XXX.XXX.XXX.XXX:9009 maxconn 128 check inter 3000 fall 2 rise 2 cookie server1
    server server2 YYY.YYY.YYY.YYY:9009 maxconn 128 check inter 3000 fall 2 rise 2 cookie server2

XXX.XXX.XXX.XXX and YYY.YYY.YYY.YYY are public IP addresses.

Apache and the vhosts configs on the web server, are configured to listen to port 9009, as per the HAProxy config. Everything is running fine up to this point.

Now, I switch on Shorewall Firewall on the web server, with a firewall rules file that looks like this:


#SECTION ALL

#SECTION ESTABLISHED

#SECTION RELATED

SECTION NEW

ACCEPT net:ZZZ.ZZZ.ZZZ.ZZZ $FW:XXX.XXX.XXX.XXX TCP 9009

ACCEPT net:WWW.WWW.WWW.WWW $FW:XXX.XXX.XXX.XXX TCP 9009

# Drop Ping from the "bad" net zone.. and prevent your log from being flooded..

Ping(DROP) net $FW

# Permit all ICMP traffic FROM the firewall TO the net zone

ACCEPT $FW net icmp


ZZZ.ZZZ.ZZZ.ZZZ is the primary HAProxy server, and WWW.WWW.WWW.WWW is the secondary HAProxy server (and not relevant right now to this problem).

And all web traffic dies when the firewall is enabled on the web server.

The HAProxy server starts sending files like these to the syslog: Message from syslogd@localhost at Sep 30 12:59:38 ... haproxy[14631]: backend www_servers has no server available!

And the web server returns 503 errors.

There are other rules in the rules file, which for instance allows SSH on a certain port from a specific IP address, and those rules all work, so I assume (although I could be wrong) it has little to do with the Shorewall firewall rules per say.

Is it so, that HAProxy needs additional ports open on the web server (this is what I suspect) and if so what would those undocumented ports be, or does anyone know what could be wrong here?

Any help would be much appreciated.

Cheers, /j.

  • 1
    With the shorewall enabled, can you connect to any of the backends (e.g. XXX.XXX.XXX.XXX:9009) directly? It seems like the issue is the shorewall, not HAproxy. – Jim G. Sep 30 '14 at 16:58
  • As it turns out (and this might be helpful to others some day) - I have fail2ban triggering bans on behalf of Shorewall, and for some strange reason, fail2ban caught HAProxy on a alleged apache-w00tw00t attack, and banned connections from the IP. So you're right, it was the firewall in the end. I've added the HAProxy server's IP to the ignoreip config of fail2ban, so that should take care of it for now. The problem I see with that is obviously that if the HAProxy server is compromised, then I'm giving potential malicious users peace and quiet to make their way onto the web server. – josa Sep 30 '14 at 18:45

It's probably the health checks that are triggering the ban, as they're not sending a hostname. You can try using 'httpchk' or 'ssl-hello-chk' instead, which might resolve the issue and allow you to re-enable fail2ban.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.