0

This morning (May 5, 2023), I found that my LAN stopped working on my Ubuntu 18.04 installation. All computers are locked out because the main server FORWARD rules are being ignored.

When I first start the computer, it works as expected. Nothing was installed (one of the first things I checked) and the forwarding has been working for like 10 years. I did not change my firewall. No installation... Just after a reboot it works for a minute or two, I think that is the time it takes for something to start and mess that up. But I would need to know where to look to determine what is causing the issue (hopefully not code from a hacker).

Below, we can see that the PREROUTING, INPUT, and POSTROUTING worked for a little bit. Now the counters are stuck at those numbers. No more packets are going through the NAT table, except the INPUT chain. What I really need, though, is the POSTROUTING chain which has a SNAT and thus generates a forward. (I also have a MASQUERADE, but that is not required, it is ignored because of the SNAT rule. I just added the MASQUERADE just in case it would help.)

Chain PREROUTING (policy ACCEPT 642 packets, 43832 bytes)
pkts    bytes target     prot opt in     out     source               destination         
 208    13747 LOG        all  --  *      *       192.168.3.183        0.0.0.0/0            LOG flags 0 level 4

Chain INPUT (policy ACCEPT 825 packets, 51359 bytes)
pkts    bytes target     prot opt in     out     source               destination         
 189     9873 LOG        all  --  *      *       192.168.0.18        0.0.0.0/0            LOG flags 0 level 4

Chain OUTPUT (policy ACCEPT 120 packets, 13320 bytes)
pkts    bytes target     prot opt in     out     source               destination         
   0        0 LOG        all  --  *      *       192.168.0.18        0.0.0.0/0            LOG flags 0 level 4

Chain POSTROUTING (policy ACCEPT 89 packets, 8970 bytes)
pkts    bytes target     prot opt in     out     source               destination         
 153    10268 LOG        all  --  *      *       192.168.0.18        0.0.0.0/0            LOG flags 0 level 4
 153    10268 SNAT       all  --  *      eno1    192.168.0.18        0.0.0.0/0            to:10.0.0.2

I show the SNAT rule. As we can see, it received the same number of packets and my MS-Windows box said that it could see the Internet for a little while. So I'm pretty sure that the network is properly setup on a reboot.

Things I've looked into already (outside of the iptables above):

$ sudo sysctl  net.ipv4.ip_forward
net.ipv4.ip_forward = 1

$ sysctl net.bridge.bridge-nf-call-iptables
net.bridge.bridge-nf-call-iptables = 1

I also checked conntrack, but there is nothing in particular I can see there, except the "UNREPLIED" which I would imagine is a different issue?

udp      17 16 src=192.168.0.18 dst=75.75.75.75 sport=56804 dport=53 ...
            ... [UNREPLIED] src=75.75.75.75 dst=192.168.0.18 sport=53 dport=56804 mark=0 use=1

Note: it is one long line, cut in half here to make it easier to read.

Also I have a docker from snap, although the daemon is running, the DOCKER rules that often appear in the firewall are not there at the moment.

Btw, the LAN itself works. I have various things that communicate with the main server or between computers and it works just fine.


The bridge addresses:

$ ip -4 -br address
lo               UNKNOWN        127.0.0.1/8 
eno1             UP             x.x.x.x/30 192.168.1.1/24 10.1.10.2/24 
eno2             UP             192.168.0.45/24 192.168.2.1/24 192.168.3.1/24 172.16.1.1/24 10.0.2.10/24 10.0.3.10/24 
docker0          DOWN           172.17.0.1/16 

And the routes:

$ ip route
default via 96.67.192.226 dev eno1 proto static 
10.0.2.0/24 dev eno2 proto kernel scope link src 10.0.2.10 
10.0.3.0/24 dev eno2 proto kernel scope link src 10.0.3.10 
10.1.10.0/24 dev eno1 proto kernel scope link src 10.1.10.2 
x.x.x.x/30 dev eno1 proto kernel scope link src x.x.x.x 
172.16.1.0/24 dev eno2 proto kernel scope link src 172.16.1.1 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 
192.168.0.0/24 dev eno2 proto kernel scope link src 192.168.0.45 
192.168.1.0/24 dev eno1 proto kernel scope link src 192.168.1.1 
192.168.2.0/24 dev eno2 proto kernel scope link src 192.168.2.1 
192.168.3.0/24 dev eno2 proto kernel scope link src 192.168.3.1 

The x.x.x.x is a static IP address.


Docker counter measure for forwarding packets:

Chain DOCKER-USER (0 references)
    pkts      bytes target     prot opt in     out     source               destination         
       0        0 early_forward  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Resolution: I upgraded to 20.04 (and will move on to 22.04 very soon). I just cannot explain why/how the 18.04 iptables would break in this way, but it really did for me. 20.04 is holding so far.

6
  • Are you sure Docker doesn't create conflicting LANs? What's the output of ip -4 -br address; ip route?
    – A.B
    May 17 at 19:43
  • @A.B Well... when I do a ps -ef | grep docker, it shows up only if it is running and at that point I see rules in my firewall. However, after a reboot and no use of docker, the daemon is not running. Also I noticed what looks like a crashed process. The name is SEPDRV_ABNORMAL and it is a kernel thread. Could that be related? May 17 at 19:50
  • (you should still obfuscate public addresses). I don't see anything obvious. You could keep a few logs running right at boot. For example: ip -ts monitor > /tmp/iproute2.log + nft monitor 2>&1 | ts > /tmp/nft.log + (for the iptables part only, even if the previous command would be enough) stdbuf -oL xtables-monitor -e | ts > /tmp/iptables.log
    – A.B
    May 17 at 19:53
  • Ah again about Docker. Docker does change the FORWARD policy to DROP, but you're aware of this right, since you're using Docker, you must have adapted your rules adequately?
    – A.B
    May 17 at 20:11
  • 1
    @A.B Well... since I really couldn't find anything close to a working solution, I decided to upgrade, since I was still on 18.04... and now that I'm on 20.04, it works. So something in 18.04 broke... The new version is holding, it passed the night and my LAN functions as before (so far). May 18 at 14:04

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.