After long searching I've found the answer to my challenge, but the last step fails.
I have two identical servers that both serve multiple guests. The servers have two interfaces: one for the external IP's I got from my ISP and one for an internal back-end network. All guests get an internal back-end network address and are connected to the physical back-end network via a bridge. Both servers can ping all guests on both servers. I expose the guests by D/SNAT'ing the guest IP to one of the available IP's on the host. This works perfectly for guests that live on the host.
For the D/SNAT to work to the other host, I learned that I needed to add a route for packets that arrived on the back-end interface, because the default route told the stack on the other server to go back on the wrong interface and wrong public IP. I have this scenario working in two local virtual test machines, even over a "dummy" bridge to the physical back-end interface.
This also works on the real servers for DNAT'ing to the back-end IP address that lives on the back-end bridge but NOT for the addresses assigned to the guests on the other server, which are also connected to the same back-end bridge.
Configuration, server A:
/etc/iproute2/rt_tables:
..
200 internal
iptables -P FORWARD ACCEPT
iptables -t nat -A POSTROUTING -o $WAN_IF -j SNAT --to $PUBLIC_IP
iptables -t nat -A PREROUTING -i $WAN_IF -p tcp -d $PUBLIC_IP --dport 53 -j DNAT --to $GUEST_IP
ip rule add from $LAN_IP table internal
ip route add default via $LAN_GW dev $LAN_IF table internal
Where
WAN_IF = WAN bridge interface (br1)
PUBLIC_IP = is public ip address on WAN bridge if
LAN_IP = private ip on LAN bridge (br0)
LAN_GW = private ip on LAN bridge on other server
brctl show
br0 8000.003018a96c83 no eth0
vnet0
vnet1
vnet2
vnet3
vnet4
vnet5
br1 8000.003018a96c84 no eth1
When $GUEST_IP is ip of guest on same host, everything works. When $GUEST_IP is LAN ip of other server also. But I can't reach guests on other server?
Update
Adjusting the default gateway of the guest works, but that isn't a symmetric solution as I wanted to create. But if that needs to be, it's a small change to make things work, as long as I keep track of the routes. It would however render routing different protocols via different public IP's impossible though unless I create some rules for that on the guest as well.
Update 2
It turns out I can solve the problem by adjusting the default gateway of the guest alone, but this is not as 'flexible' as I had hoped for.
Is there a more elegant/flexible solution?
