Port forwarding (iptables) not working from bridge network

Question

I have a server with some KVM virtual machines running using libvirt and a brige network (192.168.123.0/24). I'm trying to forward one port from the public IP of the host machine to one of the VMs (192.168.123.103). It works correctly when accessed from the internet and with one additional rule it also works from the host machine, but I can't get it to work from the machines inside the virtual network.

Here is the relevant (I think) part of the iptables configuration:

# Generated by iptables-save v1.6.0 on Tue Sep 13 16:16:26 2016
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
-A PREROUTING -d x.x.x.x/32 -p tcp -m tcp --dport 7000 -j DNAT --to-destination 192.168.123.103
-A OUTPUT -d x.x.x.x/32 -o lo -p tcp -m tcp --dport 7000 -j DNAT --to-destination 192.168.123.103
-A POSTROUTING -s 192.168.123.0/24 -d 224.0.0.0/24 -j RETURN
-A POSTROUTING -s 192.168.123.0/24 -d 255.255.255.255/32 -j RETURN
-A POSTROUTING -s 192.168.123.0/24 ! -d 192.168.123.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 192.168.123.0/24 ! -d 192.168.123.0/24 -p udp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 192.168.123.0/24 ! -d 192.168.123.0/24 -j MASQUERADE
COMMIT
# Completed on Tue Sep 13 16:16:26 2016
# Generated by iptables-save v1.6.0 on Tue Sep 13 16:16:26 2016
*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -i lo -j ACCEPT
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -m conntrack --ctstate INVALID -j DROP
-A FORWARD -i lo -j ACCEPT
-A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -m conntrack --ctstate INVALID -j DROP
-A FORWARD -d 192.168.123.103/32 -p tcp -m tcp --dport 7000 -j ACCEPT
-A FORWARD -s 192.168.123.0/24 -i virbr1 -j ACCEPT
-A FORWARD -i virbr1 -o virbr1 -j ACCEPT
COMMIT
# Completed on Tue Sep 13 16:16:26 2016

When accessing 192.168.123.103:7000 from another machine in the 192.168.123.0/24 network, I see a SYN packet from 192.168.123.36 to x.x.x.x, then the same packet with the destination replaced to 192.168.123.103, then SYN-ACK from 192.168.123.103 to 192.168.123.36 and then repeated RST packets from 192.168.123.36 to 192.168.123.103.

Does anyone have an idea how to make this work?

The only workaround I see right now is forcing DNS inside the VMs to point to the private IPs, but I'd prefer if I can keep the public DNS there.

Answer 1

First: just ignore KVM, VMs and bridge. Imagine there is no VM and no bridge anywhere. This bridge is just "the LAN" with servers inside and there's a Router (the KVM host) routing the LAN to Outside. It helps thinking about it and it's more generic.

Summarizing what happened with those packets:

LAN(L)      OUTSIDE(O)

C ----> R(B)
^     /
|    /
|   /
|  /
| /
S<

Client C (192.168.123.36) in network L sends a packet to an other IP B (x.x.x.x) in a different network O.

Router R translates the destination to Server S (192.168.123.103) in same network L and sends it back in network L. R doesn't alter source C.

S receives a packet coming from client in the same network and answers directly without the need of Router R.

C receives an unknown SYN+ACK coming from S for a never asked connection (remember C asked for B and doesn't know about S). C sends back a RST to S to abort it.

C is still waiting for a SYN+ACK coming from B, it never comes. C retries...

How to make things work? Find a way to have the flow coming back from S to be routed again by R. Router R knowns about the flow, so it will be able to handle it back to C. For S to send the packet back through R, destination musn't be in network L. Easiest configuration is using B because you know it already (and you have it already in some rules, so no complexity added). So the router R has to also change the source address from C to B when the connection is started.

The rule to change the source address in router R must be in POSTROUTING, but then the packet doesn't carry the destination B anymore (it was changed in PREROUTING already) which adds some difficulty to identify it.

Please note that the MASQUERADE rules are not part of the problem, only the DNAT and the fact that C and S are in the same LAN cause it.

Two ways to solve it:

1. Since the only way for a packet to be routed and coming back from the same interface is with the DNAT rule, assume all packets coming from L and going out to L are such packets. It's simple but there might be security considerations to double-check, especially if the rules are using IPs and never interface names.
   Append to your rules:

   iptables -t nat -A POSTROUTING -s 192.168.123.0/24 -d 192.168.123.0/24 -j SNAT --to-source x.x.x.x
   

2. Don't assume anything, mark the packet about to be DNATed in PREROUTING and use the mark to identify it later in POSTROUTING. Just note a mark follows the packet inside the router, of course not on the wire. Here it's needed and used only for the first packet, after this conntrack will handle the flow. This solution allows to work around various limitations, like wanting to use -i interface in POSTROUTING, intending to use SNAT in PREROUTING (like here)... just mark it in a first step and use the mark to do it in a 2nd step.
   So, instead, insert those rules:

   #has to be before the DNAT: DNAT is final, MARK will continue
   iptables -t nat -I PREROUTING -s 192.168.123.0/24 -d x.x.x.x/32 -p tcp -m tcp --dport 7000 -j MARK --set-mark 1
   
   iptables -t nat -A POSTROUTING -m mark --mark 1 -j SNAT --to-source x.x.x.x
   

Note: S will only see IP x.x.x.x for all clients in lan L. There are additional methods for this (eg: using NETMAP and a fake unused LAN to have each client be mapped uniquely for logging purposes)