VXLAN L3 over Wireguard L3, with VLAN-VNI Mapping

Question

Hoping this is the right place - I originally posted on Network Engineering but it got closed and I was pointed to Server Fault.

I am currently attempting to setup a L2 bridge between two sites using VXLAN to provide the L2 connectivity and Wireguard as transport/L3. I've previously done a Layer 2 bridge like this using GRE over Wireguard and it's been rock-solid, but I'm trying to better understand VXLAN now, and am looking to replace the GRE tunnel with VXLAN.

I've been trying to make use of the info both here and here but for the life of me I can't get traffic to pass over the non-wireguard IPs between sites.

I have two Debian machines with bridge-utils installed. They're also running nftables with rules to drop all DHCP traffic as when I first setup the GRE tunnel I ended up with machines getting assigned IPs from the remote network. But everything else is set to allow and it's only exposed externally via the Wireguard port

Host A is setup with:

Wireguard wg0 - 172.30.100.1/24
Bridge br0 - 10.0.0.160/24

Host B is setup with:

Wireguard wg0 - 172.30.100.2/24
Bridge br0 - 10.1.0.160/24

The AllowedIPs on the Wireguard configs is only for the Wireguard subnet 172.30.100.0/24. This was working with the GRE config and I'd assume would work with VXLAN too, as the VXLAN traffic is encapsulated within the Wireguard tunnel. The hosts can ping and ssh each other on their Wireguard IPs, so that bit is working fine.

The bridges both have port ens18, bridge-vlan-aware yes and bridge-vids 1-4096 in /etc/network/interfaces

I have a script based on the 'Recipe 2' from the first link I posted above, i.e. a single tunnel with multiple VNIs. The idea is that the script adds the VXLAN interface vx0 to br0, which waits until after wg0 is up (using a systemd service), and then loops to do the VLAN/VNI mapping.

#!/bin/bash

# Gets Wireguard interface IP address.
wgip=`ip a s wg0 | egrep -o 'inet [0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}' | cut -d' ' -f2`

ip link add vx0 type vxlan dstport 4789 external local $wgip dev wg0 # Creates vxlan with wg0 IP as local
#Here is where I may be going wrong but I've tried various combinations ^^^

sleep 1
ip link set dev vx0 master br0 # Adds vxlan to bridge
bridge link set dev vx0 vlan_tunnel on # Enables vlan tunnel on vxlan

# Maps each VLAN to VNIs across the tunnel.
for vlan in 10 20 30; do
        bridge vlan add vid $vlan dev vx0
        bridge vlan add vid $vlan dev ens18
        bridge vlan add dev vx0 vid $vlan tunnel_info id $vlan

# Think I can remove the below line if I switch to BGP-EVPN for learning later?
bridge fdb append 00:00:00:00:00:00 dev vx0 vni $vlan dst 10.1.0.160

done

ip link set dev vx0 up

I may be completely on the wrong track here, but if there's anything that looks off in the above, any guidance in the right direction would be greatly appreciated!

(It might even just be down to routing rather than config. Wireguard config is set to Table=off which I did in the GRE/WG config)

UPDATE Looks like the external flag in the vx0 setup line is causing dramas. If I change it to the below, and also change the static FDB entry to the remote wg0 IP rather than the remote ens18 IP, I can ping host to host:

ip link add vx0 type vxlan dstport 4789 vni 1000 local $wgip dev wg0

Then it looks like untagged traffic goes via VNI 1000, and VIDs 10, 20 and 30 are mapped to their respective VNIs.

What does the external flag do that would be preventing the original config from working? Is it the lack of explicit mapping of VID 1?

Answer 1

Did you ever get this resolved? I'm not attempting to bridge a trunked interface with vlan tagging like you appear to have been attempting, just normal access interface, but I don't think it would behave any differently than what I just got working since mine is bringing across all the other ethernet and IP frames like spanning tree, arp, etc. So if 802.1q is in the mix I don't think it would matter.

My setup:

server1 in data center 1. Interface ens192 is not IP'd and connected on an access port mapped to VLAN 100. ens224 is publicly IP'd and connected to local VLAN 80, but for the sake of example we'll call it 10.0.1.10/24. wireguard has a typical generic setup where this node is 192.168.0.1.

server2 in data center 2. Interface ens192 is not IP'd and connected on an access port mapped to VLAN 200. ens224 is publicly IP'd, and connected to local VLAN 90, and we'll call it 172.16.0.10/24. wireguard is 192.168.0.2.

I'm working with ubuntu+netplan, so my yaml looks like this for server 1 and same for 2 with the IP change:

network:
  ethernets:
    ens192:
      mtu: 1500
      dhcp4: false
      dhcp6: false
      accept-ra: false
      link-local: [ ]
    ens224:
      #mtu: 1400
      mtu: 1500
      addresses:
      - 10.0.1.10/24
      gateway4: 10.0.1.1
      nameservers:
        addresses:
        - 8.8.8.8
  version: 2

Wireguard config, super plain:

[Interface]
Address = 192.168.0.1/24
SaveConfig = true
ListenPort = 1240
PrivateKey = privkey

[Peer]
PublicKey = pubkey
AllowedIPs = 192.168.0.2/32
Endpoint = 172.16.0.10:1240

Apply that config, next up, install the bridge-utils package.

Finally, I set up the vxlan interface for a VNI of 150 (hey, half way between 100 and 200), used the wireguard IP's as local and remote, default port. Create the bridge interface. Bind the vlan 100 interface (ens192) to the bridge, and vxlan150 to the bridge, turn off spanning tree since it would interfere with the switches on both sides (and I could even have redundant wireguard bridges if I wanted and let STP figure it out on the network side), then bring the interfaces up:

ip link add vxlan150 type vxlan id 150 local 192.168.0.1 remote 192.168.0.2 dstport 0
brctl addbr bridge150
brctl addif bridge150 ens192
brctl addif bridge150 vxlan150
brctl stp bridge150 off
ip link set up dev vxlan150
ip link set up dev bridge150

Once I disabled firewalld packets started blasting across, with the remote VLAN 200 becoming a part of local VLAN 100. Spanning tree on the remote side saw the advertisements from the root port on the local side and went through normal reconvergence. That obviously requires a delicate touch if you're connecting two networks that both have redundant paths, local layer three forwarding, etc. since getting it wrong and sticking this link in the middle can cause a real loop and/or isolation.

Performance seems great. I just need to tinker with firewalld so I can keep it alive on the public-facing interfaces.