We've just implemented a secondary internet connection. It will be used as pure failover so for the time being I just want to avoid async routing issues.
I've setup 2 routing tables:
fw1 ~ # ip route sho tabl 10 | grep default
default via 59.167.xx.xx dev eth0 src 59.167.xx.xx
fw1 ~ # ip route sho tabl 20 | grep default
default via 115.64.xx.xx dev eth0.2 src 115.64.xx.xx
Table 10 is our primary connection, Table 20 is the new connection. I've setup the rules to select the routing table based on the connection mark:
fw1 ~ # ip rule sho
0: from all lookup local
32736: from all fwmark 0x14 lookup 20
32737: from all fwmark 0xa lookup 10
32766: from all lookup main
32767: from all lookup default
The appropriate rules are also setup in iptables:
fw1 ~ # iptables -t mangle -nvL PREROUTING
Chain PREROUTING (policy ACCEPT 300K packets, 164M bytes)
pkts bytes target prot opt in out source destination
372K 201M CONNMARK all -- * * 0.0.0.0/0 0.0.0.0/0 CONNMARK restore
371K 201M PBR all -- * * 0.0.0.0/0 0.0.0.0/0 /* this stream has no mark; send it to the PBR chain */ mark match 0x0
fw1 ~ # iptables -t mangle -nvL PBR
Chain PBR (1 references)
pkts bytes target prot opt in out source destination
10 666 MARK-gw1 all -- eth0 * 0.0.0.0/0 59.167.xx.xx/29 /* prevent asynchronous routing */ state NEW mark match 0x0
18 1128 MARK-gw2 all -- eth0.2 * 0.0.0.0/0 115.64.xx.xx/29 /* prevent asynchronous routing */ state NEW mark match 0x0
fw1 ~ # iptables -t mangle -nvL MARK-gw1
Chain MARK-gw1 (1 references)
pkts bytes target prot opt in out source destination
10 666 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 /* send via 59.167.xx.xx */ MARK set 0xa
10 666 CONNMARK all -- * * 0.0.0.0/0 0.0.0.0/0 CONNMARK save
10 666 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
fw1 ~ # iptables -t mangle -nvL MARK-gw2
Chain MARK-gw2 (1 references)
pkts bytes target prot opt in out source destination
18 1128 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 /* send via 115.64.xx.xx */ MARK set 0x14
18 1128 CONNMARK all -- * * 0.0.0.0/0 0.0.0.0/0 CONNMARK save
18 1128 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
I'm testing with a ping from an outside VPS. The connection tracking table confirms that the mark is being set correctly:
icmp 1 28 src=173.255.xx.xx dst=115.64.xx.xx type=8 code=0 id=29301 src=115.64.xx.xx dst=173.255.xx.xx type=0 code=0 id=29301 mark=20 secmark=0 use=2
But tcpdump shows the reply being sent via eth0 instead of eth0.2 even though it has the correct source address:
fw1 ~ # tcpdump -lnn -i eth0 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
14:07:56.429765 IP 115.64.xx.xx > 173.255.xx.xx: ICMP echo reply, id 29301, seq 619, length 64
