32

My home server has two main interfaces, eth1 (a standard internet connection) and tun0 (an OpenVPN tunnel). I'd like to use iptables to force all packets generated by a local process owned by UID 1002 to exit through tun0, and all other packets to exit through eth1.

I can easily mark matched packets:

iptables -A OUTPUT -m owner --uid-owner 1002 -j MARK --set-mark 11

Now, I'd like to put some rule in the POSTROUTING chain (probably of the mangle table) to match packets marked with 11 and send them to tun0, followed by a rule that matches all packets and send them to eth1.

I found the ROUTE target, but that seems to only re-write the source interface (unless I'm reading it incorrectly).

Is iptables capable of this? Do I have to mess around with the routing table (via ip route or just the legacy route commands) instead?

Edit: I thought that maybe I should provide more information. I have no other iptables rules at present (although I may create some rules to carry out unrelated tasks in the future). Also, the output of ip route is:

default via 192.168.1.254 dev eth1  metric 203
10.32.0.49 dev tun0  proto kernel  scope link  src 10.32.0.50
85.17.27.71 via 192.168.1.254 dev eth1
192.168.1.0/24 dev eth1  proto kernel  scope link  src 192.168.1.73  metric 203

I haven't touched the routing table - this is just how it is at present (although it looks fairly dirty). I'm sorry, but I don't have the legacy route command installed on this machine.

And the output of ip addr (of course, eth0 and eth2 can be ignored - they're NICs that aren't being used at present):

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 1c:6f:65:2a:73:3f brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast     state UP qlen 1000
    link/ether 00:1b:21:9d:4e:bb brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.73/24 brd 192.168.1.255 scope global eth1
    inet6 fe80::21b:21ff:fe9d:4ebb/64 scope link
       valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 00:1b:21:6a:c0:4b brd ff:ff:ff:ff:ff:ff
5: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc     pfifo_fast state UNKNOWN qlen 100
    link/none
    inet 10.32.0.50 peer 10.32.0.49/32 scope global tun0

Edit: I've gotten something sorta working, but it's not forwarding marked packets to tun0. Basically, I added a table (11), and used:

ip route add table 11 via 10.32.0.49 dev tun0
ip rule add priority 10000 fwmark 11 table 11

When I just sudo -u user1000 wget -qO- whatismyip.org, I get my home's external IP address, but if I do sudo -u user1002 wget -qO- whatismyip.org, I also get my home's IP address (but I should be getting the IP at the other end of the OpenVPN tunnel).

Running iptables -vL confirms that the packets are getting matched by the marking rule, but they don't appear to be following the routing rule.

EDIT: I've spent a long time on this, and although it still doesn't work, I think I'm a bit closer.

The iptables rule has to be in the mangle table's OUTPUT chain. I think I also need a MASQUERADE rule in the nat table's POSTROUTING chain, to set the source address. However, the re-routing that occurs after OUTPUT's mangle is not working correctly.

I've spent 5 hours on this now, so I'm taking a break and will probably return to it later tonight or sometime tomorrow.

Improve this question

5 Answers 5

Reset to default
37

I've recently hit a similar issue, albeit a slightly different. I wanted to route only TCP port 25 (SMTP) over an OpenVPN tap0 interface, while routing all other traffic (even for the same host) over the default interface.

To do so, I had to mark packets and set up rules for handling it. First, add a rule that make the kernel route packets marked with 2 through table 3 (explained later):

ip rule add fwmark 2 table 3

You could have added a symbolic name to /etc/iproute2/rt_tables, but I did not bother to do so. The number 2 and 3 are randomly chosen. In fact, these can be the same but for clarity I did not do it in this example (although I do it in my own setup).

Add a route for redirecting traffic over a different interface, assuming the gateway being 10.0.0.1:

ip route add default via 10.0.0.1 table 3

Very important! Flush your routing cache, otherwise you will not get a response back and sit with your hands in your hair for some hours:

ip route flush cache

Now, set a firewall rule for marking designated packets:

iptables -t mangle -A OUTPUT -p tcp --dport 465 -j MARK --set-mark 2

The above rule applies only if the packets come from the local machine. See http://inai.de/images/nf-packet-flow.png. Adjust it to your requirements. For instance, if you only want to route outgoing HTTP traffic over the tap0 interface, change 465 to 80.

To prevent the packets sent over tap0 getting your LAN address as source IP, use the following rule to change it to your interface address (assuming 10.0.0.2 as IP address for interface tap0):

iptables -t nat -A POSTROUTING -o tap0 -j SNAT --to-source 10.0.0.2

Finally, relax the reverse path source validation. Some suggest you to set it to 0, but 2 seems a better choice according to https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt. If you skip this, you will receive packets (this can be confirmed using tcpdump -i tap0 -n), but packets do not get accepted. The command to change the setting so packets get accepted:

sysctl -w net.ipv4.conf.tap0.rp_filter=2
Improve this answer
3
  • Why do you use SNAT instead of MASQUERADE? Doesn't MASQUERADE do the same thing, except the IP address is gathered from the interface at "runtime" and not hardcoded into the config file?
    – Ethan
    Jun 1, 2012 at 15:33
  • 8
    @Ethan From man iptables: It should only be used with dynamically assigned IP (dialup) connections: if you have a static IP address, you should use the SNAT target. Masquerading is equivalent to specifying a mapping to the IP address of the interface the packet is going out, but also has the effect that connections are forgotten when the interface goes down. You are right, but since my IP would never change, I decided to use SNAT per recommendation by the manpage.
    – Lekensteyn
    Jun 1, 2012 at 16:25
  • I think (although not tested) the last step (reverse path source validation) is not necessary. You could simply add another rule to the PREROUTING chain to DNAT to the correct source address (according to your default gateway).
    – Christian Wolf
    Sep 27, 2017 at 16:27
9

I solved this. The issue was with the routing rules in table 11. Table 11 was getting hit, but the routing rules make it inoperable. This script is what I now use, and it seems to work well (although it's obviously specific to my setup). Also, I created a new table 21 devoted to the main uplink (eth1).

# Add relevant iptables entries.
iptables -t mangle -A OUTPUT -m owner --uid-owner 1002 -j MARK --set-mark 11
iptables -t nat -A POSTROUTING -o tun0 -j MASQUERADE

# Flush ALL THE THINGS.
ip route flush table main
ip route flush table 11
ip route flush table 21
ip rule flush

# Restore the basic rules and add our own.
ip rule add lookup default priority 32767
ip rule add lookup main priority 32766
ip rule add fwmark 11 priority 1000 table 11
# This next rule basically sends all other traffic down eth1.
ip rule add priority 2000 table 21

# Restore the main table.  I flushed it because OpenVPN does weird things to it.
ip route add 127.0.0.0/8 via 127.0.0.1 dev lo
ip route add 192.168.1.0/24 dev eth1 src 192.168.1.73
ip route add default via 192.168.1.254

# Set up table 21.  This sends all traffic to eth1.
ip route add 192.168.1.0/24 dev eth1 table 21
ip route add default via 192.168.1.254 dev eth1 table 21

# Set up table 11.  I honestly don't know why 'default' won't work, or
# why the second line here is needed.  But it works this way.
ip route add 10.32.0.49/32 dev tun0 table 11
ip route add 10.32.0.1 via 10.32.0.50 dev tun0 table 11
ip route add 0.0.0.0/1 via 10.32.0.50 dev tun0 table 11

ip route flush cache

## MeanderingCode edit (because I can't comment, yet)

Thanks for this answer! It seems as though this could get messy, as you would have to maintain route info here (possibly duplicating, or breaking other things which may want to set routes.

You may be experiencing "weird things" in your routing table from OpenVPN because the server is configured to "push" routes, enabling all traffic to route through the VPN network interface, rather than the "bare" internet. Or your OpenVPN config or whatever script/application sets it up is setting routes.

In the former case, you can edit your OpenVPN configuration and put in a line containing "route-nopull"
In the latter, check configuration for OpenVPN or any wrapper (network-manager-openvpn, for example on many current linux desktop distros)
In either case, eliminating the routing configuration where it's getting set is cleaner and safer than flushing the table, depending on when you run this script and what else your system is doing.

Cheers!

Improve this answer
2

I think you want: 

/sbin/ip route add default via 10.32.0.49 dev tun0 table 11
/sbin/ip rule add priority 10000 fwmark 11 table 11

http://lartc.org/howto/lartc.netfilter.html

But I haven't tested the above.

Improve this answer
3
  • I tried it. It seems to have the same effect as leaving the default keyword out.
    – Ethan
    Dec 29, 2011 at 21:54
  • To eliminate options you might try doing a packet capture on tun0, and see if packets are ending up on that interface. tcpdump -i tun0.. alternatively, do you need to base it on user ID? Could you just do it based on the destination IP address (easier)? route add <remote host> via 10.32.0.49
    – mfarver
    Dec 29, 2011 at 21:58
  • I can't base if off the destination IP address - there would be many of those. I could also theoretically base it off pid, but that's even more difficult, because I won't know the pid until the daemon(s) start.
    – Ethan
    Dec 30, 2011 at 0:00
1

For IPv4:

1. Redirect UID-specific traffic to a separate routing table, e.g. UID 1002 to table 502:

ip rule add uidrange 1002-1002 table 502

2. Use the desired source IP in the default gateway rule in the separate routing table.

ip route add default via <gateway-ip> src <src-ip> table 502

<gateway-ip> you can find by running ip route.
<src-ip> is the outbound IP you want to use.

For IPv6:

Same process, slightly different commands:

ip -6 rule add uidrange 1002-1002 table 502
ip -6 route add default via fe80::1 dev <src-if> src <src-ip> table 502

<src-if> is the outbound network interface you want to use, e.g. eth0.
<src-ip> is the outbound IPv6 you want to use.

Improve this answer
1

This can be done without iptables command. Just execute a simple ip command

For uid 1002:

ip rule add uidrange 1002-1002 table 502

Then add a default route for this table 502 over the interface that you want, say:

ip rule add default via a.b.c.d dev eth0
Improve this answer
4
  • I tested that command, and it did not work: Error: argument "uidrange" is wrong: Failed to parse rule type
    – kasperd
    Dec 30, 2015 at 14:45
  • @kasperd you must he been doing it wrong, cos it works fine for me.
    – horseyguy
    May 18, 2019 at 9:22
  • ip rule add uidrange was added to the Linux 4.9 in 11/16 and iproute2 about 6 months later in iproute2 version 4.10.
    – John S Gruber
    Jan 9, 2020 at 19:28
  • 1
    The second command makes no sense. Did you mean ip route add default via a.b.c.d table 502? It won't work ;)
    – rustyx
    Sep 25, 2022 at 16:20

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .