I have a need for a secondary routing table on this machine. I've got everything mostly set up, but I'm having an issue getting the default route working on the secondary fib.
In most information I can find about secondary routing tables, a simple eg.
setfib 1 route add default 10.0.0.1 (where 10.0.0.1 is the router on the network) is used to set a default route for the secondary fib. In those cases, 10.0.0.1 is an external router on the network, and is a static route. In my case, 10.0.0.1 is the machine where I'm trying to configure this. It is also the router for the entire network. If I add that route, I can't connect to outside sources as the routing table pretty much loops back on itself. For example:
First, a route default route is added (a route to 10.0.0.1/16 is already in place)
# setfib 1 route add default 10.0.0.1 add net default: gateway 10.0.0.1 fib 1
Pinging the router (which is the machine itself on the primary FIB) works fine:
# setfib 1 ping 10.0.0.1 PING 10.0.0.1 (10.0.0.1): 56 data bytes 64 bytes from 10.0.0.1: icmp_seq=0 ttl=64 time=0.043 ms ^C --- 10.0.0.1 ping statistics --- 1 packets transmitted, 1 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 0.043/0.043/0.043/0.000 ms
But external sources cannot be reached - "Time to live exceeded":
# setfib 1 ping 220.127.116.11 PING 18.104.22.168 (22.214.171.124): 56 data bytes 36 bytes from this_server.my.domain (10.0.0.1): Time to live exceeded Vr HL TOS Len ID Flg off TTL Pro cks Src Dst 4 5 00 0054 24ce 0 0000 01 01 3f00 10.0.0.1 126.96.36.199 ^C --- 188.8.131.52 ping statistics --- 1 packets transmitted, 0 packets received, 100.0% packet loss
A traceroute to the external source reveals the problem:
# setfib 1 traceroute 184.108.40.206 traceroute to 220.127.116.11 (18.104.22.168), 64 hops max, 40 byte packets 1 this_server (10.0.0.1) 0.043 ms 0.036 ms 0.024 ms 2 this_server (10.0.0.1) 0.033 ms 0.030 ms 0.028 ms 3 this_server (10.0.0.1) 0.027 ms 0.031 ms 0.032 ms ...snip... 64 this_server (10.0.0.1) 0.132 ms 0.138 ms 0.137 ms
The public interface on this machine is on another physical port. If I manually add the ISP provided gateway which is assigned via DHCP, the fib works - though that's not a long term solution as the gateway can change. I'm aware I may be able to set up some sort of hook to re-assign the gateway automatically when provided with a new one, but I'd rather do this the clean and correct way.
I guess what I'm looking for is how to route traffic on that fib over the other interface, rather than over an IP? Is that even a thing? I honestly don't know enough about routing to really know where I'm meant to be going with this.
Here's an abbreviated version of the routing table on both fibs (some established OpenVPN tunnels and other misc things such as an ipv6 tunnel are omitted):
# netstat -r4 Routing tables Internet: Destination Gateway Flags Netif Expire default lo0-100.NYCMNY-VFT UGS em0 10.0.0.0/16 link#1 U re0 # setfib 1 netstat -r4 Routing tables (fib: 1) Internet: Destination Gateway Flags Netif Expire default this_server UGS re0 10.0.0.0/16 link#1 U re0
I appreciate any help at all with this!