Joining two layer-3 interfaces for failover

Question

I have a Linux machine with two USB modems each with its own SIM card and public IP address.

One of the modems may lose connectivity at some point and I want the connections to be as resilient as possible. I understand it is probably impossible to sustain existing connections, but at least once interface 1 loses connectivity, I want interface 2 to be used for new connections immediately.

What is the best way to achieve minimum downtime with two interfaces in Linux?

                                                ____________
                         +------------+        /            \
                         |            |       |              |
           +-------------+ Provider 1 +-------                \
           |             |            |     /                  \
    +------+--------+    +------------+    |                    |
    |     if1       |                      /                    \
    |               |                      |                    |
    + Linux machine |                      |     Internet       |
    |               |                      |                    |
    |     if2       |                      \                    /
    +------+--------+    +------------+    |                    |
           |             |            |     \                  /
           +-------------+ Provider 2 +-------                /
                         |            |       |              |
                         +------------+        \____________/

Answer 1

I would suggest having both interfaces up all the time and tweak your routing table is such a way that:

• Your primary interface will always be preferred
• Your secondary has an IP and is ready (hot standby) but traffic is not going through (so you don't get changed too much)
• Have a second default gateway with higher metric

You can do the above using

ip route add default dev xyz metric 100

Depending on your linux distro you will be able to make it permanent by editing the post-up or if-up scripts.

Another approach, especially if you are not getting charged per-usage, is to load balance the two interfaces. I have never setup anything similar ... but it seems pretty straight forward using lnlb. Nope my mistake, this is for LAN/cluster load-balancing, but this has an iptables based implementation.

UPDATE: Based on comment:

A routing table looks like the following:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.207.246.1    0.0.0.0         UG    0      0        0 br0
10.0.3.0        0.0.0.0         255.255.255.0   U     0      0        0 lxcbr0
10.207.246.0    0.0.0.0         255.255.255.0   U     0      0        0 br0
169.254.0.0     0.0.0.0         255.255.0.0     U     1000   0        0 br0

The default route(s) are the ones with destination 0.0.0.0. You can see mine is:

0.0.0.0         10.207.246.1    0.0.0.0         UG    0      0        0 br0

The metric indicates the priority and the lowest number is the most preferred. For example, with two interfaces I could have the following, in which case I prefer to send traffic to my br0 (bridge) interface:

0.0.0.0         10.207.246.1    0.0.0.0         UG    0      0        0 br0
0.0.0.0         10.205.243.1    0.0.0.0         UG    100    0        0 eth1

If br0 goes down for any reason, the second default route will be the only route and thus eth1 will be used. Once br0 comes back up, the preferred route will change again and point to that interface.

Now, the second approach as I understand it, uses two distinct routing tables! Using the example from the linked blog, you can see one table called bsnl and one table called tata. Each table maps to an internet provider and each one has:

1. A directly connected LAN:

   ip route add 192.168.1.0/24 dev eth1 src 192.168.1.10 table bsnl
   

2. A default route/gateway:

   ip route add default via 192.168.1.1 table bsnl
   

3. And a rule which I assume maps incoming traffic to a routing table(correct me if I am wrong - reading this):

   ip rule add from 192.168.1.10 table bsnl
   

The above configuration defined the WAN/ISP side of things (one set for each internet connection) Last, the global scope default route will decide the nexthop for traffic that is coming from the LAN and has to be routed to WAN. (Global scope I assume is applied for all the interfaces that do not fall into an ip rule):

ip route add default scope global nexthop via 192.168.1.1 dev eth1 weight 1 \
                                  nexthop via 192.168.0.1 dev eth2 weight 4

The most important difference is that the last technique uses both interfaces/connections simultaneously performing load-balancing with the given priorities. In the above example, eth1 is preferred 1/5th of the time (20%) which eth2 is preferred 4/5ths (80%). Contrary, in the first setup, using two default routes, only one connection is used at any time.

Disclaimer: I have never done the second setup, so some things might be wrong or not explained in complete detail...

Hope it helps