0

UPDATE:

After stepping away from the problem for a little bit I realized there was one thing that I could examine or test which I hadn't.

Reverting to an older version of Linux allowed me to work-around the issue, BUT still importantly raises the question: What's missing to make this work for newer versions of Linux?

Linux nas 4.19.13-1-lts #1 SMP Sun Dec 30 07:38:47 CET 2018 x86_64 GNU/Linux

Orig:

The Question: What else should I check to try and isolate and fix this so that bridged networking to LXC 'vm's work again?

For debugging this problem, I've disabled normal LXC container startup. Instead I'm running them manually with additional logging.

lxc-start --logfile=/var/log/lxc/debug.$(date +%Y%m%d-%H%M%S).log --logpriority=DEBUG -F -n containername

LAN hosts can ping/talk to HOST HOST can ping/talk to LAN hosts

Two different traces

On the bridge: as expected

tcpdump -pi bridge port 67 or port 68 or icmp[icmptype] == icmp-echo or icmp[icmptype] == icmp-echoreply

(time) IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from (mac) (oui Unknown)...

Yet it doesn't get relayed/forwarded to the physical network device:

tcpdump -i bridge_IF  port 67 or port 68 or icmp[icmptype] == icmp-echo or icmp[icmptype] == icmp-echoreply
(no dhcp packets)

The bridge looks correct

# brctl show
bridge name     bridge id               STP enabled     interfaces
br0             8000.bridgemac          no              enpNs0
                                                        enpNs0
                                                        vethXXXXXX

Everything looks mostly as expected there.

(provided by iptables-nft)
ebtables -L
Bridge table: filter

Bridge chain: INPUT, entries: 0, policy: ACCEPT

Bridge chain: FORWARD, entries: 0, policy: ACCEPT

Bridge chain: OUTPUT, entries: 0, policy: ACCEPT

nft list ruleset
table bridge filter {
        chain INPUT {
                type filter hook input priority -200; policy accept;
        }

        chain FORWARD {
                type filter hook forward priority -200; policy accept;
        }

        chain OUTPUT {
                type filter hook output priority -200; policy accept;
        }
}
table ip filter {
        chain INPUT {
                type filter hook input priority 0; policy accept;
        }

        chain FORWARD {
                type filter hook forward priority 0; policy accept;
        }

        chain OUTPUT {
                type filter hook output priority 0; policy accept;
        }
}

Also sysctl and module config items.

# cat /etc/sysctl.d/*
net.bridge.bridge-nf-call-iptables=0
kernel.unprivileged_userns_clone=1

# lsmod | grep table
nf_tables             147456  2 nft_reject_ipv6,nft_reject
nfnetlink              16384  1 nf_tables
x_tables               49152  1 ip6t_REJECT

# cat /etc/modprobe.d/*
blacklist ip_tables
blacklist iptable_filter
blacklist iptable_nat
blacklist ip6_tables
blacklist ip6table_filter
blacklist x_tables
install br_netfilter

--

So far everything looks like it should work, but it isn't.

(container) # ip addr add x.x.x.x/y dev z
(container) # ping VMHOST
Works, ping reply.
(container) # ping ROUTER-GW
Nope, Destination Host Unreachable

HOWEVER, now something really unexpected happens (rather, doesn't happen).

(VHMOST) # ping ROUTER-GW
Nope, Destination Host Unreachable
Also, the connection to the VMHOST (often) times out at this point.

This last twist seems to point towards an issue in something that I haven't checked so far, I'm not sure if anything in the Linux namespaces relates to this problem.

Looking at the LXC logs don't offer anything obvious, but I copied everything that I though MIGHT have been related. Nothing actually says it, but I do wonder if the namespaces got messed up somehow.

lxc-start VM TIMESTAMP.390 DEBUG    network - network.c:setup_hw_addr:2767 - Mac address "REDACTED" on "eth0" has been setup
lxc-start VM TIMESTAMP.393 INFO     conf - conf.c:mount_entry:2039 - No such file or directory - Failed to mount "/sys/fs/fuse/connections" on "/usr/lib/lxc/rootfs/sys/fs/fuse/connections" (optional)
lxc-start VM TIMESTAMP.510 INFO     utils - utils.c:lxc_mount_proc_if_needed:1239 - I am 1, /proc/self points to "1"
lxc-start VM TIMESTAMP.536 WARN     conf - conf.c:lxc_setup_devpts:1641 - Invalid argument - Failed to unmount old devpts instance
lxc-start VM TIMESTAMP.536 DEBUG    conf - conf.c:lxc_setup_devpts:1678 - Mount new devpts instance with options "gid=5,newinstance,ptmxmode=0666,mode=0620,max=1024"
lxc-start VM TIMESTAMP.537 INFO     conf - conf.c:setup_personality:1741 - Set personality to "0x0"
lxc-start VM TIMESTAMP.537 DEBUG    conf - conf.c:setup_caps:2550 - Dropped mac_admin (33) capability
lxc-start VM TIMESTAMP.537 DEBUG    conf - conf.c:setup_caps:2550 - Dropped mac_override (32) capability
lxc-start VM TIMESTAMP.537 NOTICE   conf - conf.c:lxc_setup:3745 - The container "VM" is set up

Linux salt 4.20.0-arch1-1-ARCH #1 SMP PREEMPT Mon Dec 24 03:00:40 UTC 2018 x86_64 GNU/Linux
# pacman -Qs lxc
local/lxc 1:3.1.0-1
  • Update: After stepping away from the problem for a little bit I realized there was one thing that I could examine or test which I hadn't. Reverting to an older version of Linux allowed me to work-around the issue, BUT still importantly raises the question: What's missing to make this work for newer versions of Linux? Linux nas 4.19.13-1-lts #1 SMP Sun Dec 30 07:38:47 CET 2018 x86_64 GNU/Linux – Michael J. Evans Jan 7 at 4:41
  • is ip forwarding enabled? – Sum1sAdmin Jan 7 at 10:32
  • It works on 4.19.13 without ip forwarding as this is a BRIDGED, not a ROUTED configuration. – Michael J. Evans Jan 9 at 19:46
1

Heh, following all links from the email to make sure other people that hits this knows what to do.

Apply: https://marc.info/?l=linux-netdev&m=154696956604748&w=2

Fixed it for me, it's to do with fq pacing

New contributor
Ian Kumlien is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
  • Thank you, I can confirm that applying that single line patch DOES make the difference and resolve the issue (for me). Hopefully it's backported to the 4.20.x series. ArchLinux had a package that made testing the a custom-kernel build easier, but it was based on 4.20.2, so I re-tested without (failed, as expected) the patch and with (appears to be working, as hoped). – Michael J. Evans yesterday

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.