DNAT using iptables only works for traffic incoming on eth0

Question

I have a machine with two interfaces that have different routeable IPv4 addresses. To return traffic on the right interface, I used this answer and comment, and it works: I can ssh into the machine using either IP address. (I don't know if this has any impact, but it might have.)

Now I added the following iptables rules for reverse proxying / DNAT using this answer. My boot script now looks like:

# eth0 is automatically brought up and gets 192.168.1.2 as IP

sudo ip link set eth1 up
sudo ip addr add 192.168.2.2/24 dev eth1
sudo ip rule add from 192.168.2.2 table frometh1
sudo ip route add default via 192.168.2.1 dev eth1 table frometh1

sudo iptables -t nat -A PREROUTING -i eth0 -p tcp -m tcp --dport 443 -j DNAT --to-destination 10.0.0.1:443
sudo iptables -t nat -A PREROUTING -i eth1 -p tcp -m tcp --dport 443 -j DNAT --to-destination 10.0.0.1:443
sudo iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
sudo iptables -t nat -A POSTROUTING -o eth1 -j MASQUERADE

echo 1 > /proc/sys/net/ipv4/ip_forward

Initially, the -i eth* was omitted and the DNAT rule was just one command. The MASQUERADE rules were also one command by exchanging -o eth* for -d 10.0.0.1. I split them in an attempt to make the interfaces explicit, in case iptables implicitly takes one of the interfaces.

When I try to reach 192.168.1.2:443, the traffic is forwarded and masqueraded as expected. The hit counter for the correct DNAT rule in the PREROUTING table is incremented and the right MASQUERADE rule is incremented. I can successfully build a TCP connection.

When trying to reach 192.168.2.2:443, I can see the traffic incoming on the interface, and the hit counter for correct DNAT rule in the PREROUTING table is incremented. But there is no outgoing packet and neither of the two MASQUERADE rules in POSTROUTING is incremented. The TCP SYN never reaches the remote system (10.0.0.1).

Note that it does not matter if the forwarding happens through eth0 or eth1, it's fine to forward the traffic to the target (10.0.0.1) over either eth0 or eth1.

Why do the iptables rules work only for eth0 and not for eth1?

Edit: It might be relevant to know that proxying in userspace works fine:

mkfifo /tmp/fifo
sudo nc -kvlp 443 </tmp/fifo | nc 10.0.0.1 443 >/tmp/fifo

Allows traffic to flow both ways, as described in this blog. The problem with this solution is that the remote end sees only one, persistent TCP connection. It should see one connection per client and handle multiple clients simultaneously.

Answer 1

First, let's draw your network topology.

I think this is side effect of the reverse path filter, because the packets are dropped at the routing decision step.

Check the output of nstat -az 'TcpExtIPReversePathFilter' command. Likely it shows nonzero counter, that are being incremented at checks.

Check the routing with command ip route get 10.0.0.1 from <external-host-ip> iif eth1. I think it will show something like invalid cross-device link.

You can see the current state of rp_filter on the interfaces in the ip netconf show command output.

Disable the rp_filter or better set it into loose mode. The rp_filter prevents a spoofing of ip addresses in downstream networks, what is widely used in the DDoS amplification attacks. In your case the tuning doesn't add the new risks because your linux host isn't the main gateway for connected networks.

It's not a last issue in your setup. Fix the problem with rp_filter if exists and I'll extend the answer to make your configuration work.

Update

Now, when the rp_filter issue has resolved, let's try to make the configuration work as expected.

Let's start from describing, what's happening and goes wrong.

1. Client host sends the packet to ext2.IP. Original packet looks like

C.IP:<someport> -> <ext2.IP>:443

2. Packet arrives to GW2 and after passing through the port forwarding will looks like shown below and will be sent to the linux host.

C.IP:<someport> -> 192.168.2.2:443

3. When the packet arrives to the linux host the DNAT rule rewrite the destination again into 10.0.0.1:443. Now we have packet in form:

C.IP:<someport> -> 10.0.0.1:443

4. Only after the destination rewriting the linux lookups the route (the routing decision). Because the source address in the packets is still original, your additional routing rule won't involved, and the packet will be routed through the main routing table (I assume the GW1 will be used). We'll improve it because the root cause of the issue is in this.

5. Packet is sent to the GW1 through eth0 interface. The source address will be rewritten by the -o eth0 -j MASQUERADE rule. The packet will be transformed into:

192.168.1.2:<otherport> -> 10.0.0.1:443 with the destination MAC address of the GW1.

6. The GW1 forwards this packet into external network with additional source address rewriting.

<ext1.IP>:<otherport2> -> 10.0.0.1:443

7. The remote system receives the packet, handle it and answer on it with the tcp datagram with flags SYN-ACK. The answer will look like:

10.0.0.1:443 -> <ext1.IP>:<otherport2>

8. The GW1 receives the answer, makes the reverse address translation of the destination address and sends the answer further to the linux host:

10.0.0.1:443 -> 192.168.1.2:<otherport>

9. Finally the linux host receives the SYN-ACK answer, makes the reverse address translation too and trying to lookup the route to forward the answer further. But after this step something goes wrong, but this is just a consequence of issue on the step 4. For better understanding need to check the firewall rule set.

10.0.0.1:443 -> <C.IP>:<someport>

The solution

The main goal is to route back the packets through the same interface, on which these packets has been received. To do it we will use the simple routing rules.

1. Create the two additional routing table (by one per uplink channel)

ip route add 192.168.1.0/24 dev eth0 table 10
ip route add 0/0 via 192.168.1.1 dev eth0 table 10

ip route add 192.168.2.0/24 dev eth1 table 11
ip route add 0/0 via 192.168.2.1 dev eth1 table 11

2. Create the routing rules with match by input interface:

ip rule add iif eth0 lookup 10 pref 1010
ip rule add iif eth1 lookup 11 pref 1011

All packets, received on the particular interface, will be routed by limited routing table, in which there are only routes for single interface. It simplest solution, but it removes the ability of the linux host forward the packets between interfaces (from eth0 to eth1 and vice versa). If that isn't suitable for you, there is another way, but it's more complex. I'll describe it if you need.