ebtables/iptables/bridge-utils: PREROUTING/FORWARD issue on single-NIC bridge

Question

We have a number of iptables rules for forwarding connections, which are solid and work well.

For example, port 80 forwards to port 8080 on the same machine (the webserver). When a given webserver is restarting, we forward requests to another IP on port 8080 which displays a Maintenance Page. In most cases, this other IP is on a separate server.

This all worked perfectly until we installed bridge-utils and changed to using a bridge br0 instead of eth0 as the interface.

The reason we have converted to using a bridge interface is to gain access to the MAC SNAT/DNAT capabilities of ebtables. We have no other reason to add a bridge interface on the servers, as they don't actually bridge connections over multiple interfaces.

I know this is a strange reason to add a bridge on the servers, but we are using the MAC SNAT/DNAT capabilities in a new project and ebtables seemed to be the safest, fastest and easiest way to go since we are already so familiar with iptables.

The problem is, since converting to a br0 interface, iptables PREROUTING forwarding to external servers is no longer working.

Internal PREROUTING forwarding works fine (eg: request comes in on port 80, it forwards to port 8080).

The OUTPUT chain also works (eg: we can connect outwards from the box via a local destination IP:8080, and the OUTPUT chain maps it to the Maintenance Server IP on a different server, port 8080, and returns a webpage).

However, any traffic coming into the box seems to die after the PREROUTING rule if the destination IP is external.

Here is an example of our setup:

Old Setup:

iptables -t nat -A PREROUTING -p tcp --dport 9080 -j DNAT --to-destination $MAINTIP:8080
iptables -a FORWARD --in-interface eth0 -j ACCEPT
iptables -t nat -A POSTROUTING --out-interface eth0 -j MASQUERADE
echo 1 > /proc/sys/net/ipv4/ip_forward

New Setup: (old setup in various formats tried as well..., trying to log eth0 and br0 packets)

iptables -t nat -A PREROUTING -p tcp --dport 9080 -j DNAT --to-destination $MAINTIP:8080
iptables -a FORWARD --in-interface br0 -j ACCEPT
iptables -t nat -A POSTROUTING --out-interface br0 -j MASQUERADE
echo 1 > /proc/sys/net/ipv4/ip_forward

Before changing to br0, the client request would go to server A at port 9080, and then be MASQUERADED off to a different server $MAINTIP.

As explained above, this works fine if $MAINTIP is on the same machine, but if it's on another server, the packet is never sent to $MAINTIP under the new br0 setup.

We want the packets to go out the same interface they came in on, MASQUERADED, as they did before we switched to using a single-NIC bridge (br0/bridge-utils).

I've tried adding logging at all stages in iptables. For some reason the iptables TRACE target doesn't work on this setup, so I can't get a TRACE log, but the packet shows up in the PREROUTING table, but seem to be silently dropped after that.

I've gone through this excellent document and have a better understanding of the flow between iptables and ebtables: http://ebtables.sourceforge.net/br_fw_ia/br_fw_ia.html

From my understanding, it seems that the bridge is not forwarding the packets out the same interface they came in, and is dropping them. If we had a second interface added, I imagine it would be forwarding them out on that interface (the other side of the bridge) - which is the way bridges are meant to work ;-)

Is it possible to make this work the way we want it to, and PREROUTE/FORWARD those packets out over the same interface they came in on like we used to?

I'm hoping there are some ebtables rules which can work in conjunction with the iptables PREROUTING/FORWARD/POSTROUTING rules to make iptables forwarding work the way it usually does, and to send the routed packets out br0 (eth0) instead of dropping them.

Comments, flames, any and all advice welcome!

Best Regards, Neale

Answer 1

When filtering on a bridge i think you need to use a different match for the interface in/out, in your case, in i understand your setup correctly, something like

iptables -t nat -A PREROUTING -p tcp --dport 9080 -j DNAT --to-destination $MAINTIP:8080
iptables -a FORWARD -m physdev --physdev-in eth0 -j ACCEPT
iptables -t nat -A POSTROUTING -m physdev --physdev-out eth0 -j MASQUERADE

I don't think you need to enable ip_forward ... this is a bridge afterall

last but not least

echo 1 > net.bridge.bridge-nf-call-iptables

make sure this is enabled , but it should by default.

Answer 2

I've faced with same issue a long time ago. There is the magic option hairpin on the bridge port interface, that implements this strange behaviour. If the hairpin disabled, then the frames won't be forwarded back through interface, through which these frames have been received.

One more interesting thing about this issue: when the interface is moved out into the promisc mode, all work perfectly. Most funny part is in that, it happens when you run the tcpdump to try to troubleshoot the problem.