I've been dealing with quite an odd issue for about the past week now, and I'm wondering if anyone had any advice to continue debugging. We have a machine with two ethernet interfaces, one of which serves as our internet connection (let's say IP A), and the other which has a subnet of our network's traffic (which includes IP A) mirrored to it through our router; that interface has IP B. We have an application that takes in traffic from the mirrored interface and filters it, dumping some packets to tun interfaces (let's simplify and only say tun0 here, though in the real application we have one for each core on which the filtering is performed). We then attempt to DNAT packets received on tun0 back to IP B on port C so that another application can ingest them.

We have a rule set up in the nat table's PREROUTING chain to perform the DNAT, as follows:

iptables -t nat -I PREROUTING -p tcp -i tun0 -j DNAT --to <B>:<C>

I have confirmed that this rule is being hit, both with packet traces as well as looking at the packet numbers from iptables. After that, however, the packets seem to disappear—I would expect them to next enter the INPUT chain as IP B is a local address (I've confirmed ip route get <B> refers to lo), however the packets don't enter the INPUT or FORWARD chains, which leads me to believe they're dropped in the routing decision. Everything that I've seen online seems to point to this being an issue with rp_filter, forwarding or route_localnet, however I've confirmed all of the following:

net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.tun0.rp_filter = 0
net.ipv4.conf.lo.rp_filter = 0
net.ipv4.ip_forward = 1
net.ipv4.conf.tun0.forwarding = 1
net.ipv4.conf.lo.forwarding = 1
net.ipv4.conf.all.route_localnet = 1
net.ipv4.conf.tun0.route_localnet = 1
net.ipv4.conf.lo.route_localnet = 1

When I turn on martian logging, I see the following:

IPv4: martian source <B> from <A>, on dev tun0

Which seems to confirm that the DNAT is having some effect, but given that rp_filter=0 this confuses me.

At this point, I'm out of ideas on how to continue debugging, as the packets seem to disappear from the face of the earth after the DNAT step other than the martian logs, and would greatly appreciate some guidance! Thanks!

  • the command tcpdump will show all packets that pass through the host regardless whether they are filtered by iptables or not. So you can see if they are really forwarded to the destination host or not. – Bodo Hugo Barwich Jul 7 at 16:43
  • tcpdump -i any port <C> doesn't seem to give me any output. – Carson Hoffman Jul 7 at 16:47
  • first check if there is any traffic coming in on your interface with tcpdump -i tun0 -XX. if the traffic comes in on your gateway but does not go to the desired port means that your rule does not work as expected. Furthermore I saw that the documentation at linux-ip.net/html/nat-dnat.html references the -j DNAT --to-destination Action. Perhaps it might be a spelling issue. – Bodo Hugo Barwich Jul 7 at 17:58
  • I've confirmed that the traffic is coming into tun0 with tcpdump and that the packets are hitting the DNAT rule (I see them hitting rule 1 in nat:PREROUTING on a trace and I see the packet counts go up for the rule in iptables -t nat -nvL). Thank you for that link; it appears that iptables will try to autocomplete the name and when using --to-destination the same result occurs (both show to:<B>:<C> in the same iptables output), but that's certainly a safer way to use that option, and I will make sure to use that going forward. – Carson Hoffman Jul 7 at 18:06

The packets that were being fed to my mirrored interface were from my local machine, and I was unaware that any packets that make it to the INPUT chain with a local source IP address would be dropped. The fact that the packet traveled to our router and back to our other ethernet interface made that less obvious (e.g. "what would make this machine any different from the other ones on our network if they all go through the router"), but the same source IP was (logically) still retained. This behavior makes sense for a "normal" use case, though I was unable to find it described in any external documentation on the subject (it could be from a lack of knowledge, but I can promise it wasn't a lack of effort); I only found it while reading through the Linux Kernel's source:

 *  NOTE. We drop all the packets that has local source
 *  addresses, because every properly looped back packet
 *  must have correct destination already attached by output routine.

Because we were dumping packets directly onto a tun interface and retaining the original source IP address, this goes against the kernel's expectations of a "normal" situation and it silently drops the packet, making tracking down the cause quite a pain. It also appears (from reading a different section of the same file) that this martian log doesn't necessarily indicate that the packet was blocked for that reason—only that it was detected as a martian. Moving forward, our solution will likely either be "don't do that" (make requests from the same machine), or figure out a way to rewrite the source IP address before the INPUT chain executes, as the usual SNAT step in the POSTROUTING chain would be too late.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.