How to make lxd containers accessible to the LAN

Question

If I use Virtualbox to spin up a VM, I can select "bridged" as the network adapter type and this results in the guest/virtual NIC being connected to my physical LAN and hence getting a LAN IP from my router (via DHCP).

I want to achieve this same functionality, but instead of using Virtualbox, I want to use lxc/lxd containers.

How can I achieve this?

Edit 1

I'm running Ubuntu and I tried following this guide:

https://insights.ubuntu.com/2015/11/10/converting-eth0-to-br0-and-getting-all-your-lxc-or-lxd-onto-your-lan/

...but it doesn't help. Should the original host interface have an IP afterwards? Because it doesn't when I try the manual bridging.

Edit 2

If it helps, my lxd/lxc host is a Virtualbox virtual machine running Ubuntu, set up with bridged networking to my physical Ethernet NIC.

Edit 3

If I use tcpdump to monitor icmp traffic on the bridge interface, the physical/host interface and the container/virtual interface, only the container/virtual does not get any traffic. The other two do.

Edit 4

According to this guide:

http://www.microhowto.info/troubleshooting/troubleshooting_ethernet_bridging_on_linux.html

I have no issues with my bridge setup.

Yet as mentioned in "Edit 3", the container interface is not getting any traffic. Need to work out why, but I'm not sure how to...

I have a feeling it has something to do with routes.

The container has no routes, whereas the host does.

Edit 5

Using tcpdump to monitor arp traffic, shows that arp traffic is actually getting to the container/virtual interface.

So it's just icmp traffic that isn't.

Edit 6

If I set a static IP in the container (via /etc/network/intefaces*), that allows me to ping the container from the host (which is a Virtualbox machine).

If I then change the network configuration in Virtualbox to allow promiscuous traffic, I can then ping the container from my physical machine (the host of the Virtualbox machine). Yet from here, I am still unable to ping beyond my physical LAN, from within the container.

The last step, if I manually add a "default" route in the container like so:

route add default gw 192.168.0.1 eth0

that allows me to ping outside of the physical LAN from inside the container.

So unless someone else can offer an explanation (I'll wait before posting an answer), I'm guessing the lack of container DHCP support (via bridging) has something to do with the fact that lxc/lxd is using netmasq to handle DHCP (and DNS).

Answer 1

1. If your LXD host is actually a virtual machine, ensure that the virtual machine's network adaptor is configured to promiscuous mode, so that way LXD container traffic passes from the physical to the virtual network.
2. Set a static IP address in the lxd container(s), because DHCP (from your physical gateway) doesn't seem to work.

In my 6th edit I said that I needed to manually add a default route in the container, but that's not true. I only needed to do that because I forgot to specify the gateway LAN IP address in the /etc/network/interfaces file. So it's not an LXD issue, just don't forget to specify it.

Answer 2

You need to create a bridge interface, which your host OS interface will be one leg of, and then attach lxc containers to this bridge - this way you will get the proper bridged connectivity.

Part of your container config will be looking like this:

lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = br0

While your OS bridge configuration will be actually distro-specific.

Answer 3

What you basically need is the containers to use a default gateway that can forward the traffic to the rest of your network, and a static route on your router that can forward back to that system. This could be your lxd host (using ip forwarding in Linux), or you could have a dedicated container for handling the routing (possibly running a firewall to limit which container IP/Ports can be accessed).

If you are using the default lxdbr0 bridge and want to change the gateway for containers you can use the raw.dnsmasq settings. It usually will be set to the IP you used for the lxd host when configuring the bridge, but can be changed using dhcp-option=3:

lxc network show lxdbr0
config:
  ipv4.address: 192.168.4.1/24
  ipv6.address: none
  raw.dnsmasq: dhcp-option=3,192.168.4.2
description: ""
name: lxdbr0
type: bridge
used_by:
- /1.0/containers/ubuntu-test
managed: true

When you then restart a container you can see the new gateway using ip route show.

Once the containers are set to use the desired default gateway, make sure that system is set to forward ip packets. At this point your containers should be able to ping other IPs on that host, but won't be able to reach anything else on the network. This is because the rest of the network does not have routing information for how to return packets destine for the container subnet.

If this is a home network behind a router, you can add a static route on your home router to tell it where to send those packets. If your lxd host is using a static IP of 192.168.0.2 and your containers are using a subnet of 192.168.4.0/24, then you would add that subnet as a static route with your host .2 host IP as the gateway. That should allow packets to be routed between your containers and local network using your lxd host as a gateway between the two networks