I have two servers running CentOS 7 with public IPs that are also connected to the same LAN. Server A has some VMs, one of which I'm trying to forward port 80 to. I'm doing this using
firewall-cmd, and my WAN NIC is joined to the external zone in both cases (LAN is internal zone). This of course means masquerade is enabled (since that's the default for external). I have also checked (at least a dozen times) to make sure that
ip_forwarding is set to 1, and that the setting persists after reboot.
Here is the output of
firewall-cmd --list-all on Server A:
external (active) target: default icmp-block-inversion: no interfaces: em1 sources: services: ports: protocols: masquerade: yes forward-ports: port=80:proto=tcp:toport=80:toaddr=192.168.11.1 source-ports: icmp-blocks: rich rules:
As far as I am aware, this should work, and yet it doesn't. To test, I run
nc -vl 80 on the VM, and
nc -v [external IP] 80 on an off-site server. The connection times out.
The same configuration on Server B works perfectly fine. I'm able to send and receive text using netcat without issue.
Here's a network diagram for some additional detail:
WAN | _____Switch_____ / \ em1 enp1s0 | | zone: external +----------+ +----------+ --------------- | Server A | | Server B | +----------+ +----------+ --------------- | | zone: internal bridge1 bridge1 | | | | em2 enp3s0 | \____Switch_____/ | vnet0 +--------+ | Web VM | +--------+
Network configuration is done via scripts with
NM_CONTROLLED=noon all interfaces for both servers. Both servers also have effectively the same configuration (obvious difference being IP addresses).
Setting SELinux to permissive changes nothing (it's enabled on Server B without issue, and Server A doesn't have any relevant denials showing up in the audit logs anyway)
I can pipe the traffic through netcat like so on Server A:
nc -vl 80 | nc -v 192.168.11.1 80, and it works just fine. I can also simply open port 80 on Server A and listen from there without issue. So it's strictly forwarding that's the problem here.
iptables -t nat -Sto view the loaded rules, and I see the following:
-A PRE_external_allow -p tcp -m mark --mark 0x66 -j DNAT --to-destination 192.168.11.1:80
When I run
iptables -t nat -vnL | grep :80I can see the packet counter increase when I try to connect (although again, the connection times out), so I know the rule is being hit.
I have tried forwarding the port to a different device on the network not located on either server, with the same results (Server B works, Server A fails).
Server A is only just now being connected to WAN for the first time, so I don't know if forwarding was ever working in the past.
As for major differences, I can only think of the following:
Both servers are running different kernel versions (Server A is on
3.10.0-693.11.1.el7.x86_64, Server B is on
3.10.0-514.26.2.el7.x86_64). However, I did try running the older kernel on Server A, and that didn't solve the issue (so I'm relatively certain this isn't a kernel bug, unless it's related to the NIC drivers, although that would still be odd).
Server A is running firewalld v0.4.4.4, while Server B is on v0.4.3.2. I haven't yet tried rolling back to v0.4.3.2 on Server A (yum can't find that version of the package).
As you may have guessed, Server B hasn't had a major update since August. As much as I should, I'm hesitant to update it at this point out of fear the forwarding will break on there as well (rest assured, I will update eventually. Hopefully soon). Unfortunately, this means that there is an exceedingly large number of packages which have newer versions on Server A. Simply doing a diff of package versions would result in a lot of info to sift through.
I also noticed that some of the iptables rules were a bit different:
-A POSTROUTING_ZONES -o em1 -g POST_external -A POSTROUTING_ZONES -o bridge1 -g POST_internal -A POSTROUTING_ZONES -o em2 -g POST_internal -A POSTROUTING_ZONES -g POST_external -A PREROUTING_ZONES -i em1 -g PRE_external -A PREROUTING_ZONES -i bridge1 -g PRE_internal -A PREROUTING_ZONES -i em2 -g PRE_internal -A PREROUTING_ZONES -g PRE_external
-A POSTROUTING_ZONES -o enp1s0 -g POST_external -A POSTROUTING_ZONES -o bridge1 -j POST_internal -A POSTROUTING_ZONES -o enp3s0 -j POST_internal -A POSTROUTING_ZONES -g POST_external -A PREROUTING_ZONES -i enp1s0 -g PRE_external -A PREROUTING_ZONES -i bridge1 -j PRE_internal -A PREROUTING_ZONES -i enp3s0 -j PRE_internal -A PREROUTING_ZONES -g PRE_external
Server B is using
jumpfor the internal zone, while Server A is using
goto. This could be due to the difference in firewalld version, and is the only thing I can think of at this point which may be causing this problem (though I still find this unlikely since this is related to the internal zone).
Both servers have docker installed and configured. Containers on Server A are strictly internal only. Some containers on Server B have port forwarding to the internet (configured via
docker run). Server B also runs OpenVPN.
The only other thing I can think of to note is that the hardware is very different between the servers (Server A is a Dell T420, Server B is LGA 775 custom, so just desktop hardware).
I'm starting to go a tad insane at this point. I've even tried enabling masquerading on the internal zone, as I've seen suggested elsewhere (which makes absolutely no sense, and surprise, didn't fix the issue).
What am I missing here?
Tried forwarding locally to a different port on Server A (i.e.
firewall-cmd --zone=external --add-forward-port=port=80:proto=tcp:toaddr=192.168.11.3:toport:81). I can connect on port 80 when listening with netcat on Server A on port 81, so local forwarding is working fine. Just can't forward to other IP addresses.