Issue with multiple bridging for KVM hosts

Question

I'm using KVM and libvirt on my host (Debian lenny) + 2 bridges per guest (one for mgmt, one for public traffic).
That setup isn't stable at all, sometimes I can do pings to a management ip, sometimes not.
I don't know if my bridging paramateres are correct, could you check ? or if there is anything wrong ...

Please also note that interface on guest doesn't flap and that I got not logs on my host.
Of course forwarding is enabled.

iface eth3 inet manual

auto bond0
iface bond0 inet manual
slaves eth1 eth2
pre-up ip link set bond0 up
down ip link set bond0 down

auto br0
iface br0 inet static
address 10.160.0.7
netmask 255.255.255.128
bridge_ports eth3
bridge_fd 9
bridge_hello 2
bridge_maxage 12
bridge_stp off

auto br0:1
iface br0:1 inet static
address 10.160.0.9
netmask 255.255.255.128

auto br0:2
iface br0:2 inet static
address 10.160.0.10
netmask 255.255.255.128

auto br1
iface br1 inet static
address 217.4.40.242
netmask 255.255.255.240
gateway 217.4.40.241
pre-up /etc/network/firewall start
bridge_ports bond0
bridge_fd 9
bridge_hello 2
bridge_maxage 12
bridge_stp off

auto br1:1
iface br1:1 inet static
address 217.4.40.252
netmask 255.255.255.240

auto br1:2
iface br1:2 inet static
address 217.4.40.253
netmask 255.255.255.240

Answer 1

You have multiple interfaces with addresses on the same subnet with similar netmasks. The odds are a different route is being used for different packets, causing confusion. If you look at the output of

ip route show

You've probably got multiple default gateways as a result. For each set of IPs on the same subnet, give one of them the netmask you have, and the rest a netmask of 255.255.255.255. Those other interfaces will then only be used for packets originating or destined for that address and not others on the subnet.

Answer 2

I've a working configuration but without bonding like you (this may be the problem). Here is mine :

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo br0 br1
iface lo inet loopback

# DMZ
iface br0 inet static
    address 192.168.100.1
    netmask 255.255.255.0
    gateway 192.168.100.254
    network 192.168.100.0
    broadcast 192.168.100.255
    bridge_ports eth0
    bridge_fd 9
    bridge_hello 2
    bridge_maxage 12
    bridge_stp off

# Internal
iface br1 inet static
    address 192.168.0.1
    netmask 255.255.255.0
    gateway 192.168.0.254
    network 192.168.0.0
    broadcast 192.168.0.255
    bridge_ports eth1
    bridge_fd 9
    bridge_hello 2
    bridge_maxage 12
    bridge_stp off

Do you have error messages for bond in your /var/log/syslog message or dmesg ?

Answer 3

I've had the same issue trying to use KVM guests with 2 interfaces on the same subnet. I can reach the management interface if I start up the guest with the other interface's cable unplugged, but as soon as I plug in the 2nd nic it goes haywire.

One solution I've see mentioned is to specify VLANs when creating the interfaces on the guest. These aren't 802.1x VLANs, rather they're QEMU-specific broadcast domains that will keep network traffic from bouncing around (and possibly looping) between bridged interfaces on the same subnet.

Problem is that (except in OpenSolaris) there is currently no way to do this directly with the virt-install or virsh tools, or even to dump/edit/reload the domain's xml file to assign VLANs. The only way I'm aware of to do this is by using the kvm command directly to create guests, but it's much more involved than virt-install. I'm still trying to sort out how to translate the options from a virt-install installation into qemu-kvm, but will update if it works.