Blocking SSH Brute force from a HAProxy LB

Question

We're trying to grant access to an internal SSH server using a HAProxy setup on a public DMZ. This works as expected however connections into the server are originated from the HAProxy (tproxy won't work as the HAProxy is not the default route for the server which is buried in a different subnet). We wish to block brute force connections using fail2ban, denyhosts or similar.

Is there any way to block anywhere along the chain; the SSH server knows about the failed login but can't block without blocking every incoming connection. The HAProxy can descriminate but doesn't know to, syslog shipping the logs to the HAproxy might work however there doesn't appear to be a clean way to marry the HAProxy and sshd logs. Mitigating controls like rate limits on the HAProxy are already in place.

Whats the best way to handle brute force connections in this scenario?

Answer 1

The way I do this is to use policy based routing with a normal NAT setup on both firewalls.

You have two firewalls: fw1 is your normal default route firewall, fw2 is your ssh inbound firewall.

The server has two IP addresses assigned to it (on eth0 and eth0:1) the server's default route points at eth0 and goes out fw1, the ip address on eth0:1 is used to select a different routing table which has a default route that points at fw2.

The extra rules I use to set this up are like this:

ip route add default via 192.168.42.251 table vpn
ip rule add fwmark 1 table vpn
ip rule add from 192.168.42.24 table vpn

The 192.168.42.24 address is assigned to eth0:1
The machine fw1 is at 192.168.42.1
The machine fw2 is at 192.168.42.251

The DNAT rules in fw2's nat iptable must forward the connection to the 192.168.42.24 address not the server's eth0 address (which is 192.168.42.10).

As I recall the fwmark line isn't needed for basic inbound usage, but it allows me to use 'mangle' rules in iptables to specify that some connections go out fw2 automatically.

The end result is that the server always sees the connections coming from their real source IP addresses so things like source address logging and fail2ban work correctly without modification.