2

I'm trying to set up a high availability OpenVPN cluster for worker nodes that have a fixed ip regardless of which OpenVPN server they connect to. This is a requirement, as things are bound specifically to that ip address. I was planning to have a distributed etcd on each master node where I push which master node has which worker nodes (connect/disconnect script with OpenVPN) and update then update the routing table of each master accordingly (by watching the etcd key).

Everything will be running in docker containers.

I've made a diagram of the situation I have now with 3 master nodes (M1,M2,M3), 3 worker nodes (w1,w2,w3) and "sidecar" containers (M2M1, M3M1, M3M2) to connect each master to the others masters.

With the following routing tables set up everything works fine but it requires updating the routes on both the master nodes and the sidecar containers. Aside from the standard iptables -A FORWARD rules to ensure traffic is forwarded between interfaces I only had to add a single iptables rule on the worker nodes, e.g iptables -A POSTROUTING -o tap0 -m iprange --dst-range 5.0.0.0-5.255.255.255 -j SNAT --to-source 5.0.0.1 -t nat for w1, otherwise it would send the packets with the primary ip address.

network situation

I was wondering if it is possible to set up policy based routes on the sidecar containers to just have the rule "if it's coming from tap0 and it's from an ip in 5.0.0.0/8 then put it on eth0 towards the master node inside the same subnet" and the other way around "if it's coming from eth0 and it's in an ip in 5.0.0.0/8 then put it on tap0 towards the vpn gateway". If I could do that then I just need to manipulate the routes on the master nodes

Am I correct that I would need to mark the packets with iptables and then setup 2 route tables, one corresponding to each mark and then set up the route on that table?

I tried setting up a bridge before but I couldn't get it to work.

(And if there is a much better way of setting up a vpn cluster with fixed IPs than my rather convoluted setup do let me know)

EDIT:

With the help of the answer below I tested the following statements for M2M1, which works fine:

# Create the 2 tables to add specific routes on
echo "2     toeth" >> /etc/iproute2/rt_tables
echo "3     totap" >> /etc/iproute2/rt_tables

# Everything coming from eth0 will be going to the totap table and everything from tap0 will be going to the toeth table
ip rule add table totap iif eth0
ip rule add table toeth iif tap0

# Add the routes but on the specific table
ip r r 5.0.0.0/8 via 192.168.1.1 table totap
ip r r 5.0.0.0/8 via 172.30.2.2 table toeth

EDIT2:

In case anyone is interested playing around with it, I've set up a github repo

2

You can use ip rule (see man ip-rule) to setup specific routing tables based on source or destination IP and based on source or destination interface. It should be able to achieve what you want.

SELECTOR := [ from PREFIX ] [ to PREFIX ] [ iif STRING ] [ oif STRING ] ...
  • Thanks! You helped me on the right track, after all the googling I couldn't see the forest for the trees anymore. – drake7707 Aug 11 '18 at 15:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.