Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm very new to the networking world (most of my time is spent as a developer), so I could very well be missing something simple.

TL;DR version: I get 'martian packet' errors on response traffic when routing traffic through a GRE tunnel based on user, using iptables MARK and the fwmark IP rule. I can disable RP filtering but don't want to unless I have to, and I'd like to understand why it's happening in the first place.


Let's say I have a private network with the following setup:

  • a "default" gateway with an IP of 10.1.120.1
  • another gateway with an IP of 10.1.120.2
    • This gateway essentially acts as a tunneling proxy; it has a GRE tunnel interface set up with a local IP in the tunneled network of 172.16.3.1. Clients will send internet-bound requests (e.g. the destinations are public IPs) through the tunnel, and responses are sent back through the tunnel. For the purposes of this discussion, assume it just passes everything through.
  • a Linux client with...
    • one physical interface (eth0) with a private IP on the network of 10.1.120.101
    • a GRE tunnel interface (gretunl) with a private IP on the virtual/tunneled network of 172.16.3.4

The tunnel itself works great; I can easily route traffic generated on the client box through either eth0/default gateway, or through gretunl/tunneled gateway and get the results I'd expect.

Now, however, I want to route traffic for a particular user (tunuser) through the tunnel, and all others through eth0/default gateway. Using iptables, ip rule, and ip route, I was able to get very close.

> ip route show
default via 10.1.120.1 dev eth0
10.1.120.0/24 dev eth0  scope link
172.16.3.0/24 dev gretunl  scope link

> ip route show table tunroute
default via 172.16.3.1 dev gretunl

> ip rule show
0:      from all lookup local
32765:  from all fwmark 0x1 lookup tunroute
32766:  from all lookup main
32767:  from all lookup default

> iptables -t nat -L -v
# snipped empty chains
Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
   25  1494 SNAT       all  --  any    any     anywhere             anywhere             mark match 0x3 owner UID match tunuser to:172.16.3.4

> iptables -t mangle -L -v
# snipped empty chains
Chain OUTPUT (policy ACCEPT 22887 packets, 2857K bytes)
 pkts bytes target     prot opt in     out     source               destination
   92  7626 MARK       all  --  any    any     anywhere             anywhere             mark match 0x0 owner UID match tunuser MARK set 0x1
   92  9834 MARK       all  --  any    any     anywhere             anywhere             mark match 0x3 owner UID match tunuser MARK set 0x7

Chain POSTROUTING (policy ACCEPT 22887 packets, 2857K bytes)
 pkts bytes target     prot opt in     out     source               destination
   92  7626 MARK       all  --  any    any     anywhere             anywhere             mark match 0x1 owner UID match tunuser MARK set 0x3

I'll try to explain:

  1. The only way that I could find to route traffic by user was to mark the packet using the mangle table, then use the fwmark ip rule. So if I see a packet owned by tunuser in the OUTPUT chain, and it's not already marked, I set the mark to 0x1.
  2. The packet then goes through a routing decision, where the ip rule and tunroute table indicate that it should go through the tunnel.
  3. For some reason (possibly a routing decision made prior to the OUTPUT chain in step 1?), even after the packet is routed through the tunnel, the source IP is the IP address of the default interface/route (10.1.120.101); if I left things alone here, the "inner" packet of the tunnel would have this as the source IP instead of the correct IP for the virtual network of the tunnel (172.16.3.4). So, in the POSTROUTING chain, if the packet's mark is 0x1, I SNAT it to the correct IP. I change the mark to 0x3 so that when the encapsulated packet (which inherits the mark) comes through, I can differentiate it from a not-yet-encapsulated packet
  4. The packet goes through the tunnel interface. When the encapsulated packet (marked with 0x3) hits the OUTPUT chain, set the mark to 0x7 so that the outer destination IP doesn't get SNAT'ed.
  5. The encapsulated packet hits eth0 and is sent to the tunnel gateway.

And this all almost works. If tunuser sends an HTTP request to a public IP address, initial TCP traffic reaches the tunnel gateway and comes back - but I get Martian packet errors in the kernel logs, and the connection is broken

IPv4: martian source 10.1.120.101 from 173.194.115.48, on dev gretunl

I know I can get rid of this by disabling return path filtering using "net.ipv4.conf.all.rp_filter=0" in /etc/sysctl.conf - but I don't want to disable RP filtering unless I truly need to, and it doesn't make sense to me why it would be necessary in this case. Looking at wireshark captures, all the tunneled traffic over eth0 appears to be correctly encapsulated, e.g.

outgoing:
[IP HEADER: src=10.1.120.101, dst=10.1.120.2]
[GRE HEADER]
[IP HEADER: src=172.16.3.4, dst=173.194.115.48]
...

incoming:
[IP HEADER: src=10.1.120.2, dst=10.1.120.101]
[GRE HEADER]
[IP HEADER: src=173.194.115.48, dst=172.16.3.4]
...

Similarly, captures from gretunl show the correct src/dest on outgoing and incoming packets (same as above but without the outer IP and GRE headers). So it seems like by the time the traffic is actually reaching the gretunl interface, it has been decapsulated and thus wouldn't have a source of 10.1.120.101 anymore. I can understand why that address would be considered martian (since the tunneled interface is on the 172.16.3.0 network) - but I don't see why the tunnel interface would ever see traffic with the 10.1.120.101 destination address to begin with if decapsulation is working.

I feel like it must have something to do with either a) the fact that packets aren't being routed to the tunnel until after the first pass of the OUTPUT chain (after, I believe, a routing step that occurs between the "local process" and OUTPUT chain), or b) the fact that I'm doing SNAT on the first packet (which, again, is needed for the inner packet to have the correct 172.16.3.4 address).

Would greatly appreciate any help in understanding this, and/or ways I could accomplish this goal in a more straightforward manner. Specifically, if there were a way to route traffic by user without having to mark the packet (or to mark the packet before the initial routing decision prior to the OUTPUT chain), I could probably avoid some of the SNAT'ing and multiple markings.

share|improve this question
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.