UPDATE:
After stepping away from the problem for a little bit I realized there was one thing that I could examine or test which I hadn't.
Reverting to an older version of Linux allowed me to work-around the issue, BUT still importantly raises the question: What's missing to make this work for newer versions of Linux?
Linux nas 4.19.13-1-lts #1 SMP Sun Dec 30 07:38:47 CET 2018 x86_64 GNU/Linux
Orig:
The Question: What else should I check to try and isolate and fix this so that bridged networking to LXC 'vm's work again?
For debugging this problem, I've disabled normal LXC container startup. Instead I'm running them manually with additional logging.
lxc-start --logfile=/var/log/lxc/debug.$(date +%Y%m%d-%H%M%S).log --logpriority=DEBUG -F -n containername
LAN hosts can ping/talk to HOST HOST can ping/talk to LAN hosts
Two different traces
On the bridge: as expected
tcpdump -pi bridge port 67 or port 68 or icmp[icmptype] == icmp-echo or icmp[icmptype] == icmp-echoreply
(time) IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from (mac) (oui Unknown)...
Yet it doesn't get relayed/forwarded to the physical network device:
tcpdump -i bridge_IF port 67 or port 68 or icmp[icmptype] == icmp-echo or icmp[icmptype] == icmp-echoreply
(no dhcp packets)
The bridge looks correct
# brctl show
bridge name bridge id STP enabled interfaces
br0 8000.bridgemac no enpNs0
enpNs0
vethXXXXXX
Everything looks mostly as expected there.
(provided by iptables-nft)
ebtables -L
Bridge table: filter
Bridge chain: INPUT, entries: 0, policy: ACCEPT
Bridge chain: FORWARD, entries: 0, policy: ACCEPT
Bridge chain: OUTPUT, entries: 0, policy: ACCEPT
nft list ruleset
table bridge filter {
chain INPUT {
type filter hook input priority -200; policy accept;
}
chain FORWARD {
type filter hook forward priority -200; policy accept;
}
chain OUTPUT {
type filter hook output priority -200; policy accept;
}
}
table ip filter {
chain INPUT {
type filter hook input priority 0; policy accept;
}
chain FORWARD {
type filter hook forward priority 0; policy accept;
}
chain OUTPUT {
type filter hook output priority 0; policy accept;
}
}
Also sysctl and module config items.
# cat /etc/sysctl.d/*
net.bridge.bridge-nf-call-iptables=0
kernel.unprivileged_userns_clone=1
# lsmod | grep table
nf_tables 147456 2 nft_reject_ipv6,nft_reject
nfnetlink 16384 1 nf_tables
x_tables 49152 1 ip6t_REJECT
# cat /etc/modprobe.d/*
blacklist ip_tables
blacklist iptable_filter
blacklist iptable_nat
blacklist ip6_tables
blacklist ip6table_filter
blacklist x_tables
install br_netfilter
--
So far everything looks like it should work, but it isn't.
(container) # ip addr add x.x.x.x/y dev z
(container) # ping VMHOST
Works, ping reply.
(container) # ping ROUTER-GW
Nope, Destination Host Unreachable
HOWEVER, now something really unexpected happens (rather, doesn't happen).
(VHMOST) # ping ROUTER-GW
Nope, Destination Host Unreachable
Also, the connection to the VMHOST (often) times out at this point.
This last twist seems to point towards an issue in something that I haven't checked so far, I'm not sure if anything in the Linux namespaces relates to this problem.
Looking at the LXC logs don't offer anything obvious, but I copied everything that I though MIGHT have been related. Nothing actually says it, but I do wonder if the namespaces got messed up somehow.
lxc-start VM TIMESTAMP.390 DEBUG network - network.c:setup_hw_addr:2767 - Mac address "REDACTED" on "eth0" has been setup
lxc-start VM TIMESTAMP.393 INFO conf - conf.c:mount_entry:2039 - No such file or directory - Failed to mount "/sys/fs/fuse/connections" on "/usr/lib/lxc/rootfs/sys/fs/fuse/connections" (optional)
lxc-start VM TIMESTAMP.510 INFO utils - utils.c:lxc_mount_proc_if_needed:1239 - I am 1, /proc/self points to "1"
lxc-start VM TIMESTAMP.536 WARN conf - conf.c:lxc_setup_devpts:1641 - Invalid argument - Failed to unmount old devpts instance
lxc-start VM TIMESTAMP.536 DEBUG conf - conf.c:lxc_setup_devpts:1678 - Mount new devpts instance with options "gid=5,newinstance,ptmxmode=0666,mode=0620,max=1024"
lxc-start VM TIMESTAMP.537 INFO conf - conf.c:setup_personality:1741 - Set personality to "0x0"
lxc-start VM TIMESTAMP.537 DEBUG conf - conf.c:setup_caps:2550 - Dropped mac_admin (33) capability
lxc-start VM TIMESTAMP.537 DEBUG conf - conf.c:setup_caps:2550 - Dropped mac_override (32) capability
lxc-start VM TIMESTAMP.537 NOTICE conf - conf.c:lxc_setup:3745 - The container "VM" is set up
Linux salt 4.20.0-arch1-1-ARCH #1 SMP PREEMPT Mon Dec 24 03:00:40 UTC 2018 x86_64 GNU/Linux
# pacman -Qs lxc
local/lxc 1:3.1.0-1
