Sorry if this has been asked before, but I've been researching this for days and still can't work out the best approach.
I have 2 Ubuntu 18.04 Servers in separate physical locations both running a series of KVM VMs. I also have several VPN server's located in other countries.
Originally tried creating a libvirt virtual network on a KVM server that has a VPN server as the default gateway. That way and machines I connect to that interface are transparently isolated from the local environment.
I have been looking at many different ways to solve this, virtual networks, namespaces, even a(or several) pfsense VMs to route this. I even considered the viability of using docker containers to manage each VPN link and bind that to a tun interface?? I like the idea of being able to backup, remove and recreate the configuration without interfering with existing network functionality.
For the VPN element I've considered several options: Zerotier - I already have Zerotier setup on all hosts and is mainly used as an access layer for SSH, but I noticed it can also configured to provide Vpn functionality with a default gateway. I have experimented with using a Zerotier network with a default gateway configured, then forwarding traffic through the zt0 interface on the server. Below is the output of 'virsh net-edit zt2nat'. Using this network I can ping through to hosts on the zerotier network from the VM guest but can't ping google.com (this works when connecting via other machines using the zerotier client directly).
<network> <name>zt2nat</name> <uuid>b43030dc-e462-447d-a315-1191960db5cf</uuid> <forward dev='zt2' mode='nat'> <interface dev='zt2'/> </forward> <bridge name='virbr1' stp='on' delay='0'/> <mac address='52:54:00:75:76:b5'/> <domain name='zt2nat'/> <ip address='192.168.30.1' netmask='255.255.255.0'> <dhcp> <range start='192.168.30.128' end='192.168.30.254'/> </dhcp> </ip> </network>
Shadowsocks - I have Shadowsocks proxy setup on all VPN servers and this was my previous solution (using a Shadowsocks client on each of the VM guests) but I think using a VPN instead of a socks proxy for this would be best. binding each shadowsocks port to a TUN/TAP device may be an option.
Wireguard - I've been wanting to find an excuse to implement Wireguard for a while, in a similar way to what I tried with Zerotier, bridging the wg0 interface to some form of virtual network NAT that could provide dhcp for the VM guests and forward all traffic over wg0 to the VPN server.
The solution I choose for this would need to be scalable, as there are often 30+ VMs on one server and several VPN server endpoints. New VM's are regularly created for testing purposes and destroyed so as this system has expanded, setting up VPN clients on guest VMs each time has become tiresome. There will be at least 10 vpn servers, and therefore will need 10 Interfaces/virtual networks on the server. Trying to work out how to implement this for so many interfaces using iptables is daunting, my knowledge of networking is too limited
I've created a scaled-down rudimentary Network Diagram to explain what I'm trying to achieve, the coloured lines represent the VPN tunnels which would obviously be routed through the NAT router, but it was really difficult to draw that way.
Any general recommendations for how you would tackle this would be greatly appreciated. Or if you can spot what it is that isn't allowing my virtual network to use my zt2 interface as an upstream gateway. My knowledge of networking is limited, and days of research and learning has left me even more confused. Thanks