IPv6 connectivity lost on KVM guest after 20 minutes

Question

I have a KVM virtualization server setup at Hetzner. Hetzner provides me with a master IP (95.xxx.xxx.235) and a /29 IPv4 subnet (95.xxx.xxx.184/29) and a /64 IPv6 network (2a01:xxxx:xxxx:xxxx::/64).

The KVM Guest (Debian Stretch) loses IPv6 connectivity exactly after 20 minutes of either reboot on networking services restart. Even though the connection is lost, I can ping the default gateway (fe80::1). The IPv4 connectivity stays up all the time and has no issues.

Currently the interface is set as a macvlan in bridge mode, and I've also tried VEPA and private modes with no luck. Also the NIC type is set to e1000, but I've also tried virtio with no luck.

After connection loss I've taken a TCP dump from the physical NIC on the host, and it shows that there is echo requests leaving and echo replies arriving the interface, but a tcpdump taken from the guests NIC I could see only the requests leaving the NIC.

/etc/network/interfaces on Host:

auto lo
iface lo inet loopback
iface lo inet6 loopback

auto enp2s0
iface enp2s0 inet static
  address 95.xxx.xxx.235
  netmask 255.255.255.192
  gateway 95.xxx.xxx.193
  up route add -net 95.xxx.xxx.192 netmask 255.255.255.192 gw 95.xxx.xxx.193 dev enp2s0

iface enp2s0 inet6 static
  address 2a01:xxx:xxx:xxx::2
  netmask 64
  gateway fe80::1

/etc/network/interfaces on Guest:

auto lo
iface lo inet loopback
iface lo inet6 loopback

auto ens3
iface ens3 inet static
    address 95.xxx.xxx.187
    netmask 255.255.255.248
    gateway 95.xxx.xxx.185

iface ens3 inet6 static
    address 2a01:xxx:xxx:xxx::20
    netmask 64
    gateway fe80::1

# route -6 -n on Host:

Kernel IPv6 routing table
Destination                    Next Hop                   Flag Met Ref Use If
2a01:xxxx:xxxx:xxxx::/64          ::                         U    256 8  1162 enp2s0
fe80::/64                      ::                         U    256 0     0 macvtap0
fe80::/64                      ::                         U    256 0     0 enp2s0
::/0                           fe80::1                    UG   1024 8  4534 enp2s0
::/0                           ::                         !n   -1  1 11069 lo
::1/128                        ::                         Un   0   9    81 lo
2a01:xxxx:xxxx:xxxx::/128         ::                         Un   0   1     0 lo
2a01:xxxx:xxxx:xxxx::2/128        ::                         Un   0   9    82 lo
fe80::/128                     ::                         Un   0   1     0 lo
fe80::/128                     ::                         Un   0   1     0 lo
fe80::/128                     ::                         Un   0   1     0 lo
fe80::xxxx:xxxx:xxxx:1069/128   ::                         Un   0   1     0 lo
fe80::xxxx:xxxx:xxxx:22e1/128   ::                         Un   0   1     0 lo
fe80::xxxx:xxxx:xxxx:201/128   ::                         Un   0   2    79 lo
ff00::/8                       ::                         U    256 0     0 macvtap0
ff00::/8                       ::                         U    256 0     0 enp2s0
::/0                           ::                         !n   -1  1 11069 lo

# route -6 -n on Guest:

Kernel IPv6 routing table
Destination                    Next Hop                   Flag Met Ref Use If
2a01:xxxx:xxxx:1414::/64          ::                         U    256 0     0 ens3
fe80::/64                      ::                         U    256 0     0 ens3
::/0                           fe80::1                    UG   1024 2    77 ens3
::/0                           ::                         !n   -1  1  6846 lo
::1/128                        ::                         Un   0   5   525 lo
2a01:xxxx:xxxx:xxx::20/128       ::                         Un   0   3    70 lo
fe80::xxxx:xxxx:xxx:22e1/128   ::                         Un   0   2     6 lo
ff00::/8                       ::                         U    256 0     0 ens3
::/0                           ::                         !n   -1  1  6846 lo

# ip -6 neigh on Host:

2a01:xxxx:xxxx:xxxx::20 dev enp2s0  FAILED
fe80::1 dev enp2s0 lladdr xx:xx:xx:8d:22:06 router STALE

# ip -6 neigh on Guest:

fe80::1 dev enp2s0 lladdr xx:xx:xx:8d:22:06 router REACHABLE

Probably relevant things from /etc/sysctl.conf on Host:

net.ipv4.ip_forward=1
net.ipv4.conf.enp2s0.send_redirects=0
net.ipv6.conf.all.forwarding=1

Probably relevant things from /etc/sysctl.conf on Guest:

net.ipv6.conf.default.accept_ra=2
net.ipv6.conf.default.autoconf=1
net.ipv6.conf.all.accept_ra=2
net.ipv6.conf.all.autoconf=1
net.ipv6.conf.ens3.accept_ra=2
net.ipv6.conf.ens3.autoconf=1

Probably relevant section of the Guest libvirt-config:

<interface type='direct' trustGuestRxFilters='yes'>
  <mac address='xx:xx:xx:xx:xx:xx'/>
  <source dev='enp2s0' mode='bridge'/>
  <model type='e1000'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>

As I've been struggling with this for about two weeks and read almost every relevant post with similar issues I've seen that Hetzner apparently does some kind of fishy IPv6 implementations. I've already contacted them, but they wondered that I would have a routing problem myself. That could be true, since after 20 minutes I still receive the echo replies on the physical NIC, even though they aren't forwarded to Guest.

So, any ideas from fellow IPv6'ers?

Update:

So Hetzner confirmed me, that 2a01:xxxx:xxxx:xxxx::/64 network is routed to the link-local address of the physical interface. When restarting networking, the NDP entry stays in for 20 minutes but does get removed afterwards as the VM does not answer with correct link-local address, as it has different MAC-address.

It is starting to seem like that I cannot use macvtap-interfaces here, but I have to create a bridge for this. However, I wonder why the host cannot see the guest (and vice versa) with IPv6, when the IPv4 still works. I think it would allow me to route traffic straight from the main link-local address.

Answer 1

I had the same problem with Hetzner server but using VirtualBox instead of KVM.

Problem:

Hetzner routes all IPv6 pakets having any target IP within your /64 subnet to the MAC address of your physical host. That means that if you send a ping from somewhere in the internet to your VM that has an IPv6 address with the same prefix as the host, Hetzer's gateway does not do a neighbor solicitation to look up the MAC address of your VM but instead simply forwards the ICMP pakets to the MAC of your host. That is the reason why you can see echo replies on your physical host but not on your VM: it is targeted to your host's MAC, not the VM's MAC.

However, there seems to be a bug in Hetzner's IPv6 implementation (or it may be done on purpose, i don't know): If the VM sends a neighbor solicitation to look up the MAC address of the gateway (fe80::1) and it uses it's global IPv6 IP as the source address of the solicitation, somehow Hetzner's gateway seems to update it's internal IPv6-to-MAC address table. For the following 20 minutes, Hetzner's gateway will then send any paket targeted to the VMs IPv6 address to the VMs MAC address. If within 20 minutes no further solicitation from VM's MAC and VM's global IP is sent to the gateway, it falls back to send IPv6 pakets to the host's MAC.

Now your VM - right after network startup, maybe because the link-local address is not assigned at this point - ONCE sends a solicitation using it's global IPv6 address as the source and so "accidentially" updates Hetzner's MAC address table for once. During runtime, the VM still continuously sends solicitations to look up the MAC address of the gateway to keep it's MAC address table up to date but it uses it's link-local IPv6 address for doing so (which is totally ok from the IPv6 perspective) but that wont update Hetzner's gatway's MAC address table. That is the reason why IPv6 seems to be fully working after VM startup but only for 20 minutes.

Solutions:

There is a dirty solution and a clean solution:

• Dirty solution: Your VM has to send a solicitation for the gateway's MAC address using it's global IPv6 address from time to time (lets say every 5 minutes). This is tricky: your VM will send solicitations but using it's link-local IPv6. So i used a cheap trick here: I remove the link-local IP from the interface, send a solicitation (which is then forced to use the global IP) and re-attach the link-local IP:

  ip -6 addr del fe80::a00:27ff:fecf:e270/64 dev enp0s3
  ndisc6 fe80::1 enp0s3
  ip -6 addr add fe80::a00:27ff:fecf:e270/64 dev enp0s3
  

• Clean solution: Do not use bridging. I now use Host-only-Networking. That means the VM is connected to a separate NIC (vboxnet0). I added an IPv6 route that forwards all traffic from host to VM's IPv6 address:

  ip -6 route add <my IPv6 pefix>::20 dev vboxnet0
  

On the VM i use the link-local IPv6 address of the host as it's default GW. In order to allow the host to connect the VM on it's global IPv6 IP, i assigned another IP out of the same /64 subnet to vboxnet0. For me, this works perfectly.

Answer 2

I had exactly the same problem with my Hetzner server, but found a better solution than your "dirty solution", because ndisc6 has a parameter -s to specify another source IP instead of the link-local address:

ndisc6 -s \<my IPv6 prefix\>::20 fe80::1 enp1s0

So I've written a cron job which sends this neighbor solicitation every 5 minutes, now I'm fine :)

Answer 3

Bulding on f0ssie's answer, I have built a bash script that does the same thing but does not rely on any hardcoded addresses or interfaces for an easier deployment across multiple machines and networks. Note that aside from ndisc6, you also need iproute2 and jq.

#!/bin/bash

set -euo pipefail

default_route=$(ip -6 --json route | jq --exit-status '[.[] | select(.dst == "default")] | first')
default_gateway=$(echo "${default_route}" | jq --raw-output --exit-status '.gateway')
net_device=$(echo "${default_route}" | jq --raw-output --exit-status '.dev')
our_global_addr=$(ip -6 --json addr show dev "${net_device}" scope global | jq --raw-output --exit-status 'first | .addr_info | first | .local')

ndisc6 --source "${our_global_addr}" "${default_gateway}" "${net_device}"

Answer 4

I found that sysctl -w net.ipv6.conf.all.forwarding=1 helped. Don't know why. Inspired by https://ipv6-first-guide.hillbrecht.de/#_preparing_the_network_settings_of_the_host