I'm trying to get Linux bonding working over a VPN (GRE-TAP). The funny thing is, that it only works when I have
tcpdump running on both hosts, but more on that later...
There are two machines, called
pxn2. They are connected together using a simple switch via eth1.
pxn1 has IP address 10.1.1.197 pxn2 has IP address 10.1.1.199
To get a secure connection, all IP traffic is encrypted using IPsec. This works, I can ping between the two machines without any problem and
tcpdump shows only encrypted packets.
A GRE-TAP (tunnels Ethernet frames over IP) interface is then set up in both directions, because I will need a virtual network interface later on:
ip link add vpn_gre_pxn2 type gretap local 10.1.1.197 remote 10.1.1.199 dev eth1
vpn_gre_pxn2 Link encap:Ethernet HWaddr 1a:73:32:7f:36:5f inet6 addr: fe80::1873:32ff:fe7f:365f/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1462 Metric:1 RX packets:19 errors:0 dropped:0 overruns:0 frame:0 TX packets:26 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:1294 (1.2 KiB) TX bytes:1916 (1.8 KiB)
This is on
pxn1. On the other host the same interface is set up in the other direction.
A bridge is set up that currently uses only the GRE-TAP device.
I need the bridge because later on I want to add more machines (my plan is to bridge all GRE tunnels together). The end result should become a VPN mesh network (with a dedicated GRE-TAP interface for each host-host combination). But since for now I'm just doing a first test with two machines, the bridge is of course somewhat useless, but nonetheless important for the test itself.
brctl addbr vpn_br brctl addif vpn_br vpn_gre_pxn2
The bridge works because when I activate the
vpn_br interface and set up some IP addresses (just for testing the bridge), ICMP PINGs work perfectly.
vpn_br Link encap:Ethernet HWaddr 02:00:0a:01:01:c5 UP BROADCAST RUNNING SLAVE MULTICAST MTU:1462 Metric:1 RX packets:11 errors:0 dropped:0 overruns:0 frame:0 TX packets:6 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:448 (448.0 B) TX bytes:468 (468.0 B)
A Linux Bonding interface is now set up. Again, since this is just a first proof of concept test, I'll only add a single slave to the bond.
Later on there will also be a real separate Gbit NIC with a dedicated switch that will act as the primary slave (with the VPN being just a backup), but for now the bonding interface will use the VPN only.
modprobe bonding mode=1 miimon=1000 ifconfig bond0 hw ether 02:00:0a:01:01:c5 # some dummy MAC ifconfig bond0 up ifconfig bond0 mtu 1462 ifenslave bond0 vpn_br # as said, only a single slive at the moment ifconfig bond0 172.16.1.2/24 up
The other host is set up as 172.16.1.1/24 with HWaddr 02:00:0a:01:01:c7.
This results in a theoretically working bonding interface:
bond0 Link encap:Ethernet HWaddr 02:00:0a:01:01:c5 inet addr:172.16.1.2 Bcast:172.16.1.255 Mask:255.255.255.0 inet6 addr: fe80::aff:fe01:1c5/64 Scope:Link UP BROADCAST RUNNING MASTER MULTICAST MTU:1462 Metric:1 RX packets:11 errors:0 dropped:0 overruns:0 frame:0 TX packets:6 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:448 (448.0 B) TX bytes:468 (468.0 B)
The status also looks good to me:
# cat /proc/net/bonding/bond0 Ethernet Channel Bonding Driver: v3.6.0 (September 26, 2009) Bonding Mode: fault-tolerance (active-backup) Primary Slave: None Currently Active Slave: vpn_br MII Status: up MII Polling Interval (ms): 1000 Up Delay (ms): 0 Down Delay (ms): 0 Slave Interface: vpn_br MII Status: up Speed: Unknown Duplex: Unknown Link Failure Count: 0 Permanent HW addr: 1a:73:32:7f:36:5f Slave queue ID: 0
...as does the routing table:
# ip route show 192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.2 172.16.1.0/24 dev bond0 proto kernel scope link src 172.16.1.2 10.1.1.0/24 dev eth1 proto kernel scope link src 10.1.1.197 default via 10.1.1.11 dev eth1
eht0 is a separate active NIC (Ethernet cross cable) but that should not matter IMHO.
The setup looks good to me, however, PING does not work (this was run on
# ping 172.16.1.1 PING 172.16.1.1 (172.16.1.1) 56(84) bytes of data. From 172.16.1.2 icmp_seq=2 Destination Host Unreachable From 172.16.1.2 icmp_seq=3 Destination Host Unreachable From 172.16.1.2 icmp_seq=4 Destination Host Unreachable
tcpdump on the other machine (
# tcpdump -n -i bond0 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on bond0, link-type EN10MB (Ethernet), capture size 65535 bytes 17:45:13.013791 ARP, Request who-has 172.16.1.1 tell 172.16.1.2, length 28 17:45:13.013835 ARP, Reply 172.16.1.1 is-at 02:00:0a:01:01:c7, length 28 17:45:14.013858 ARP, Request who-has 172.16.1.1 tell 172.16.1.2, length 28 17:45:14.013875 ARP, Reply 172.16.1.1 is-at 02:00:0a:01:01:c7, length 28 17:45:15.013870 ARP, Request who-has 172.16.1.1 tell 172.16.1.2, length 28 17:45:15.013888 ARP, Reply 172.16.1.1 is-at 02:00:0a:01:01:c7, length 28
However, when I also run
pxn1 in a separate terminal, I suddenly get my ICMP replies!
... From 172.16.1.2 icmp_seq=19 Destination Host Unreachable From 172.16.1.2 icmp_seq=20 Destination Host Unreachable 64 bytes from 172.16.1.1: icmp_req=32 ttl=64 time=0.965 ms 64 bytes from 172.16.1.1: icmp_req=33 ttl=64 time=0.731 ms 64 bytes from 172.16.1.1: icmp_req=34 ttl=64 time=1.00 ms 64 bytes from 172.16.1.1: icmp_req=35 ttl=64 time=0.776 ms 64 bytes from 172.16.1.1: icmp_req=36 ttl=64 time=1.00 ms
This only works as long as both machines have
tcpdump running. I can start/stop
tcpdump and consistenly only see replies while the program is running on both machines at the same time. It doesn't matter on which machine I try.
Is this a kernel bug or (more probable) has my configuration some problem?
Is it normal, that the bridge and bonding interface both show the same MAC address? I only configure it manually for the bonding interface, which apparently changes also the bridge..
FYI, config overview:
I get a working setup when I set the bridge interface into promiscous mode (
ifconfig vpn_br promisc
). I'm not quite sure if that is normally needed. OTOH I don't think it has any downsides...
BTW, there a similar RedHat bug report exists, but setting
bond0 down/up doesn't help in my case..