6

Today and yesterday, my server automatically rebooted and failed to bring up the network device during boot. If I reboot the machine again, then it starts up fine, I've also not encountered any issues with this during the past 2 months.

The only error logs I can find relating to this are:

Aug 23 06:37:14 server systemd[1]: Started ifup for ens16.
Aug 23 06:37:14 server systemd[1]: ifup@ens16.service: Main process exited, code=exited, status=1/FAILURE

and

Aug 23 06:37:14 server sh[281]: iptables-restore: line 10 failed
Aug 23 06:37:14 server systemd[1]: ifup@ens16.service: Main process exited, code=exited, status=1/FAILURE
Aug 23 06:37:14 server sh[281]: run-parts: /etc/network/if-pre-up.d/iptables exited with return code 1
Aug 23 06:37:14 server sh[281]: ifup: failed to bring up ens16

/etc/network/if-pre-up.d/iptables contains:

#!/bin/sh
/sbin/iptables-restore < /etc/iptables.up.rules

/etc/iptables.up.rules contains:

*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [896:90530]
-A INPUT -i lo -j ACCEPT
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p tcp --dport 80 -j ACCEPT
-A INPUT -p tcp --dport 443 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-port-unreachable
COMMIT

What could possibly be going wrong with this in an intermittent fashion, and how can I make sure it doesn't happen again?

  • Start by saving the output of /sbin/iptables-restore to a file and see what messages it contains. – RalfFriedl Aug 23 '18 at 17:32
  • Finally happened again. There was no message. The file is empty, after having changed the command to /sbin/iptables-restore < /etc/iptables.up.rules > /var/log/iptables-output. – Sam Bull Oct 2 '18 at 13:10
  • 1
    Try /sbin/iptables-restore < /etc/iptables.up.rules >& /var/log/iptables-output - ifthere's an error, it's likely to appear on STDERR. NB if your sh is really sh, not bash, modify accordingly. – MadHatter Oct 21 '18 at 11:54
  • 1
    What Linux distribution is this? – Michael Hampton Oct 21 '18 at 13:19
  • @MadHatter Assumed stderr is what was already in the error logs shown above. But, will try that out. – Sam Bull Oct 21 '18 at 15:09
2
+50

I still suspect that two executions of /etc/network/if-pre-up.d/iptables are running at the same time throughout the boot process. Because of systemd normal behavior of starting things concurrently unless advised not to do so, I believe the boot process triggers one script process for the lo interface and another for the ens16 interface. That would result in a concurrent execution of iptables-restore, which may cause errors such as iptables-restore: line 10 failed. I am unable to supply evidences though.

I am used to managing CentOS and Red Hat systems. Once upon a time, one of such servers failed to initialize iptables service on boot because systemd was starting ip6tables concurrently. That specific error is documented here: https://bugzilla.redhat.com/show_bug.cgi?id=1477413

I suggest you to handle concurrency in your script, for example, by using flock:

#!/bin/sh
/usr/bin/flock /run/.iptables-restore /sbin/iptables-restore < /etc/iptables.up.rules

Alternatively, you could check the actual value of ${IFACE} variable before restoring iptables rules (reference: man 5 interfaces):

#!/bin/sh
if [ "${IFACE}" == ens16 ]; then
    /sbin/iptables-restore < /etc/iptables.up.rules
fi

Additionally, if you just want to load iptables rules at boot time, I suggest you to use iptables-persistent instead:

# apt-get install iptables-persistent netfilter-persistent
# mv -v /etc/iptables.up.rules /etc/iptables/rules.v4
# systemctl enable netfilter-persistent.service
# rm -v /etc/network/if-pre-up.d/iptables
  • I added an echo "${IFACE}" to the script and rebooted, and it seems that it is in fact running 3 times, for --all, ens16 and lo. So, I'll try your suggestions, and hope that fixes the problem. – Sam Bull Oct 24 '18 at 12:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.