KVM: Need multiple Virtual Machines to share a VLAN'ed interface

Question

I'm not sure how to configure my Linux server which has a single VLAN'ed interface to support a number of Virtual machines which I want to bridge to my network (so that all their services appear to come from a number of real machines on my network). Do I simply create a single bridged network interface br0, tied to my existing VLAN'ed device, and have all the VMs use that interface? Or do I need to set up br0, br1, br2, etc. for each VM? Can I even tie those bridge interfaces to a VLAN'ed interface? I suppose I need to say VLAN="yes" in the bridge's ifcfg file?

True, I could simply try it and see, but at this point I am so unsure about how it all comes together that I'd like to perform a sanity check before I get a sanity wreck :-) .

I have set up a CentOS7 host:

# uname -a
Linux cha028 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
# cat /etc/redhat-release 
CentOS Linux release 7.2.1511 (Core) 

It has a single interface which has a VLAN. So hardware interface em1 has:

# cat ifcfg-em1
NAME="em1"
DEVICE="em1"
ONBOOT="yes"
TYPE="Ethernet"
BOOTPROTO="none"
HWADDR=14:fe:b5:d6:07:cc
NM_CONTROLLED=no

...and I have a ifcfg-em1.144 that has all the IP addressing in it and such. I have installed a plurality of VMs on the box. Now I need to make the network work.

How do I do it? Simply create ifcfg-br0 as per https://www.banym.de/linux/centos/setup-bridge-device-on-centos (with its own unique IP address), and simply add a line to my ifcfg-em1.144 file which says BRIDGE=br0?

And, having done so, can I attach all my VMs to the br0 device?

Thank you for your indulgence, and your help.

Here is my ifcfg-em1.144 file. Note that /etc/sysconfig/network contains the GATEWAY line:

VLAN="yes"
DEVICE="em1.144"
PHYSDEV="em1"
TYPE="Ethernet"
BOOTPROTO="static"
DEFROUTE="yes"
ONBOOT="yes"
IPADDR="10.144.101.28"
PREFIX="24"
NM_CONTROLLED=no

Answer 1

If you bridge to the vlan interface (as opposed to the hardware interface), the bridged packets from your VMs will be tagged as they are bridged out onto the wire. If you bridge to the hardware interface, all traffic will be bridged regardless of tag and you will have to move your vlan interface to be off the bridge (base interface vlan is now meaningless). Any number of VMs can share a bridge.

Typically, you would handle the vlans at the host so the guests don't have to manually configure vlans. A number of different permutations of this sounds like they would work for you.

Answer 2

Here is what I did to get the bridged network set up on my Host machine over a VLAN:

1. Physical Device - the thing that is real. The only thing it knows is that it's an Ethernet device and it has a MAC address. Oh, and get that damn Network Manager out of its face:

   (file == /etc/sysconfig/network-scripts/ifcfg-em1)

   DEVICE="em1"
   TYPE=Ethernet
   HWADDR=14:fe:b5:d6:07:cc
   BOOTPROTO=none
   ONBOOT="yes"
   NM_CONTROLLED=no
   

2. VLAN device - My vlan here is numbered 144. YMMV. You tell CentOS/Redhat's network this simply by suffixing the device name with a dot followed by a number; the ifup script looks for that and extracts the VLAN ID from the string. And you tell it it is a VLAN using the VLAN=yes directive:

   (file == /etc/sysconfig/network-scripts/ifcfg-em1.144)

   DEVICE=em1.144
   TYPE=Ethernet
   BOOTPROTO=none
   ONBOOT=yes
   VLAN=yes
   BRIDGE=br0
   NM_CONTROLLED=no
   

3. Bridged device - the Bridge is the Keeper of the IP (tm). This holds the layer 3 (IP address) information... that is, DNS, netmask, etc. For me, I like to put the default gateway in /etc/sysconfig/network. And I like to keep that ZEROCONF (169.254.x.y) IP addressing out of my routing table. So we have:

   (file == /etc/sysconfig/network-scripts/br0)

   DEVICE="br0"
   TYPE="Bridge"
   BOOTPROTO="static"
   DEFROUTE="yes"
   ONBOOT="yes"
   IPADDR="10.144.1.12"
   NETMASK=255.255.255.0
   DNS1="10.144.1.101"
   DOMAIN="example.com"
   NM_CONTROLLED=no
   DELAY=0
   

   (file == /etc/sysconfig/network):

   GATEWAY=10.144.1.1
   NOZEROCONF=true
   

Note that after network is started it may take 30-60 seconds for your switch to actually accept the packets (ie, Cisco may be configured to do this). So if you see "Destination host unreachable" when you ping your gateway, wait a minute. It may just take a while to come up.

Once the host is set up and working (ie, you can ping local gateway, then other hosts on the network, then other hosts in your infrastructure... all by IP mind you... then you can ping/traceroute/telnet-into-port-22 by hostname), then you are ready to create networking on your VMs.

This is trivially easy. You just need to ensure that they are using br0 as their network device. If you use virt-install, like I did, you can do this:

virt-install --name monitor0 --memory 2048 --disk /dev/vm_group0/thin_lv_monitor0  --cdrom ./CentOS-7-x86_64-Everything-1511.iso --network bridge=br0

Note the --network bridge=br0.

If you are using qemu and you build the virtual host without networking, or with NAT, you can correct it. Just bring the host down. Then edit /etc/libvirt/qemu/<vm name>.xml. Look for a section called interface. Change it to follow this example:

<interface type='bridge'>
  <mac address='52:54:00:22:29:cc'/>
  <source bridge='br0'/>
  <model type='virtio'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>

...That is, you will probably need to change the first interface line and the source line only (that's what I did, after having built the machines without specifying networking [it sets up NAT by default]).

Save the file, then restart libvirtd using your favorite method: systemctl restart libvirtd

Restart the VM using virsh start <vm name>. Go into your VM using its console, and edit the network as appropriate. Remember, your VM is now a bona fide member of your network so you are going to give it an IP address in the same subnet as the bridged interface br0.

I have set up two VMs sharing the VLAN'ed br0 so I am a happy camper!