This morning (May 5, 2023), I found that my LAN stopped working on my Ubuntu 18.04 installation. All computers are locked out because the main server
FORWARD rules are being ignored.
When I first start the computer, it works as expected. Nothing was installed (one of the first things I checked) and the forwarding has been working for like 10 years. I did not change my firewall. No installation... Just after a reboot it works for a minute or two, I think that is the time it takes for something to start and mess that up. But I would need to know where to look to determine what is causing the issue (hopefully not code from a hacker).
Below, we can see that the
POSTROUTING worked for a little bit. Now the counters are stuck at those numbers. No more packets are going through the NAT table, except the INPUT chain. What I really need, though, is the
POSTROUTING chain which has a
SNAT and thus generates a forward. (I also have a
MASQUERADE, but that is not required, it is ignored because of the
SNAT rule. I just added the
MASQUERADE just in case it would help.)
Chain PREROUTING (policy ACCEPT 642 packets, 43832 bytes) pkts bytes target prot opt in out source destination 208 13747 LOG all -- * * 192.168.3.183 0.0.0.0/0 LOG flags 0 level 4 Chain INPUT (policy ACCEPT 825 packets, 51359 bytes) pkts bytes target prot opt in out source destination 189 9873 LOG all -- * * 192.168.0.18 0.0.0.0/0 LOG flags 0 level 4 Chain OUTPUT (policy ACCEPT 120 packets, 13320 bytes) pkts bytes target prot opt in out source destination 0 0 LOG all -- * * 192.168.0.18 0.0.0.0/0 LOG flags 0 level 4 Chain POSTROUTING (policy ACCEPT 89 packets, 8970 bytes) pkts bytes target prot opt in out source destination 153 10268 LOG all -- * * 192.168.0.18 0.0.0.0/0 LOG flags 0 level 4 153 10268 SNAT all -- * eno1 192.168.0.18 0.0.0.0/0 to:10.0.0.2
I show the SNAT rule. As we can see, it received the same number of packets and my MS-Windows box said that it could see the Internet for a little while. So I'm pretty sure that the network is properly setup on a reboot.
Things I've looked into already (outside of the iptables above):
$ sudo sysctl net.ipv4.ip_forward net.ipv4.ip_forward = 1 $ sysctl net.bridge.bridge-nf-call-iptables net.bridge.bridge-nf-call-iptables = 1
I also checked conntrack, but there is nothing in particular I can see there, except the "UNREPLIED" which I would imagine is a different issue?
udp 17 16 src=192.168.0.18 dst=220.127.116.11 sport=56804 dport=53 ... ... [UNREPLIED] src=18.104.22.168 dst=192.168.0.18 sport=53 dport=56804 mark=0 use=1
Note: it is one long line, cut in half here to make it easier to read.
Also I have a docker from snap, although the daemon is running, the DOCKER rules that often appear in the firewall are not there at the moment.
Btw, the LAN itself works. I have various things that communicate with the main server or between computers and it works just fine.
The bridge addresses:
$ ip -4 -br address lo UNKNOWN 127.0.0.1/8 eno1 UP x.x.x.x/30 192.168.1.1/24 10.1.10.2/24 eno2 UP 192.168.0.45/24 192.168.2.1/24 192.168.3.1/24 172.16.1.1/24 10.0.2.10/24 10.0.3.10/24 docker0 DOWN 172.17.0.1/16
And the routes:
$ ip route default via 22.214.171.124 dev eno1 proto static 10.0.2.0/24 dev eno2 proto kernel scope link src 10.0.2.10 10.0.3.0/24 dev eno2 proto kernel scope link src 10.0.3.10 10.1.10.0/24 dev eno1 proto kernel scope link src 10.1.10.2 x.x.x.x/30 dev eno1 proto kernel scope link src x.x.x.x 172.16.1.0/24 dev eno2 proto kernel scope link src 172.16.1.1 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 192.168.0.0/24 dev eno2 proto kernel scope link src 192.168.0.45 192.168.1.0/24 dev eno1 proto kernel scope link src 192.168.1.1 192.168.2.0/24 dev eno2 proto kernel scope link src 192.168.2.1 192.168.3.0/24 dev eno2 proto kernel scope link src 192.168.3.1
The x.x.x.x is a static IP address.
Docker counter measure for forwarding packets:
Chain DOCKER-USER (0 references) pkts bytes target prot opt in out source destination 0 0 early_forward all -- * * 0.0.0.0/0 0.0.0.0/0
Resolution: I upgraded to 20.04 (and will move on to 22.04 very soon). I just cannot explain why/how the 18.04 iptables would break in this way, but it really did for me. 20.04 is holding so far.
ip -4 -br address; ip route?
ps -ef | grep docker, it shows up only if it is running and at that point I see rules in my firewall. However, after a reboot and no use of docker, the daemon is not running. Also I noticed what looks like a crashed process. The name is
SEPDRV_ABNORMALand it is a kernel thread. Could that be related?
ip -ts monitor > /tmp/iproute2.log+
nft monitor 2>&1 | ts > /tmp/nft.log+ (for the iptables part only, even if the previous command would be enough)
stdbuf -oL xtables-monitor -e | ts > /tmp/iptables.log