I'm trying to set up a high availability OpenVPN cluster for worker nodes that have a fixed ip regardless of which OpenVPN server they connect to. This is a requirement, as things are bound specifically to that ip address. I was planning to have a distributed etcd on each master node where I push which master node has which worker nodes (connect/disconnect script with OpenVPN) and update then update the routing table of each master accordingly (by watching the etcd key).
Everything will be running in docker containers.
I've made a diagram of the situation I have now with 3 master nodes (M1,M2,M3), 3 worker nodes (w1,w2,w3) and "sidecar" containers (M2M1, M3M1, M3M2) to connect each master to the others masters.
With the following routing tables set up everything works fine but it requires updating the routes on both the master nodes and the sidecar containers.
Aside from the standard iptables -A FORWARD rules to ensure traffic is forwarded between interfaces I only had to add a single iptables rule on the worker nodes, e.g
iptables -A POSTROUTING -o tap0 -m iprange --dst-range 22.214.171.124-126.96.36.199 -j SNAT --to-source 188.8.131.52 -t nat for w1, otherwise it would send the packets with the primary ip address.
I was wondering if it is possible to set up policy based routes on the sidecar containers to just have the rule
"if it's coming from tap0 and it's from an ip in 184.108.40.206/8 then put it on eth0 towards the master node inside the same subnet" and the other way around
"if it's coming from eth0 and it's in an ip in 220.127.116.11/8 then put it on tap0 towards the vpn gateway". If I could do that then I just need to manipulate the routes on the master nodes
Am I correct that I would need to mark the packets with iptables and then setup 2 route tables, one corresponding to each mark and then set up the route on that table?
I tried setting up a bridge before but I couldn't get it to work.
(And if there is a much better way of setting up a vpn cluster with fixed IPs than my rather convoluted setup do let me know)
With the help of the answer below I tested the following statements for M2M1, which works fine:
# Create the 2 tables to add specific routes on echo "2 toeth" >> /etc/iproute2/rt_tables echo "3 totap" >> /etc/iproute2/rt_tables # Everything coming from eth0 will be going to the totap table and everything from tap0 will be going to the toeth table ip rule add table totap iif eth0 ip rule add table toeth iif tap0 # Add the routes but on the specific table ip r r 18.104.22.168/8 via 192.168.1.1 table totap ip r r 22.214.171.124/8 via 172.30.2.2 table toeth
In case anyone is interested playing around with it, I've set up a github repo