Confusion Post FedoraCore Upgrade: NAT / port forwarding trouble, and POSTROUTING MASQUERADE has unexpected influence on forwarding ports

Question

Following a gateway / firewall system's hardware failure, a younger version of Fedora Core (17) was installed on new hardware, and the old 'iptables' and system-config-firewall files from /etc/sysconfig were used (and nothing else). Old version of FC is unknown but it was probably 14 or 15, old disk is no longer readable. (I've put the former iptables file contents below.)

This systems' job is EXCLUSIVELY to:

1. Accept port 22 (ssh) from any network source for local delivery (sshd).
2. Forward port 222 to port 22 (ssh) on an internal system, retaining the originating IP address.
3. Forward port 25 (email) to an internal system, retaining the originating IP address.
4. Forward all (non port 22) inbound traffic from the internal network to the external one, applying NAT / Masquerading along the way, and permitting associated externally generated return packets to pass through, as appropriate.

The steps outlined above are the ONLY job of this box and it was formerly working fine.

Where I started:

When returned to service, all connections to port 22 (sshd) work fine, and external connections to all other ports forward through or are dropped as appropriate, but the IP address seen by internal systems was that of the gateway rather than retaining their source IP address.

When I put the system on the wire, all heck breaks loose regarding email; the mail server system behind the gateway is rejecting mail based on IP address / system name mismatches, and eliminating the check makes the spam filter system go crazy.

Was MASQUERADING wrong interface BUT...

Sharp eyed serverfault participant DavidSchwartz noted that masquerading was turned on for the INTERNAL interface (via -A POSTROUTING -o eth0 -j MASQUERADE), explaining the internal systems seeing the gateway IP address, BUT when changed to reflect the external interface (eg: -A POSTROUTING -o eth1 -j MASQUERADE), then ALL FORWARDING STOPS! Yes, I do mean port forwarding, from the external to the internal (port 222 to port 22 on an internal system).

I really don't understand that.

For clarity, it appears that having masquerading turned on for the INTERNAL interface, while not what was intended, DOES have the positive attribute of permitting port forwarding to actually happen from the outside to the inside. Switched to be masquerading the (correct) EXTERNAL interface somehow disables port forwarding as there were NO other changes.

SNAT vs Masquerading

As there are two external facing "server systems" that also have internal interfaces which had been configured to be alternative out-bound paths, I looked at them and found they had a very different masquerading technique; there was no line like that quoted two paragraphs above, but rather this instead:

-A POSTROUTING -s 192.168.0.0/24 -o eth0 -j SNAT --to-source <external.v4.ip.addr>  

Great - I understand it, makes sense, I try that! I commented out the MASQUERADE line and included (in the same position) a SNAT line as noted here, and YAY! It did two things:

1. I get outbound network address translation as desired, and;
2. It now port-forwards inbound connect requests to port 222 from the outside to port 22 on an internal system, retaining the originating IP address as desired.

Remaining Problem

What does not work is the otherwise identically forwarded port 25 (which is to remain port 25 on the internal system).

Presently THIS is the problem I must solve! (And the Sooner The Better!) Why does the one port forward and not the other? Is it not a 'tcp' type port, and if not, what should replace it? (I didn't think it was UDP...)

Less important; it'd be nice to know why the old configuration worked but it won't now! Differences in Fedora Core? Updates to IP tables? Was I relying on bugs? -smile-

Here are the two port-forwarding lines in iptables:

-A PREROUTING -i eth0 -p tcp --dport 25 -j DNAT --to-destination 192.168.0.64:25
-A PREROUTING -i eth0 -p tcp --dport 222 -j DNAT --to-destination 192.168.0.64:22

I tried it with and without the trailing ":25" on the upper one. Again, external port 222 forwards to internal system port 22 just fine, port 25 does not forward at all.

Note that telnet access to port 25 from a system that can see the mail server system directly (not through a gateway / firewall) works perfectly. Also, I tried forwarding port 24 to port 25 as a test and got the same result (no response).

Here's the iptables file:

# eth0 is internal
# eth1 is external

*nat
:PREROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]

# NO packets get through without this next line:
-A POSTROUTING -o eth0 -j MASQUERADE
# the above line was commented out and replaced with:
-A POSTROUTING -s 192.168.0.0/24 -o eth0 -j SNAT --to-source <external.v4.ip.addr>

-A PREROUTING -i eth1 -p tcp --dport 25 -j DNAT --to-destination 192.168.0.64:25

-A PREROUTING -i eth1 -p tcp --dport 222 -j DNAT --to-destination 192.168.0.64:2
2

COMMIT
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -i eth0 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT

-A INPUT -m state --state NEW -m tcp -p tcp --dport 25 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 222 -j ACCEPT

-A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT
-A FORWARD -p icmp -j ACCEPT
-A FORWARD -i lo -j ACCEPT
-A FORWARD -i eth0 -j ACCEPT
-A FORWARD -o eth0 -j ACCEPT
-A FORWARD -i eth1 -m state --state NEW -m tcp -p tcp -d 192.168.0.64 --dport 25
 -j ACCEPT
-A FORWARD -i eth1 -m state --state NEW -m tcp -p tcp -d 192.168.0.64 --dport 22
 -j ACCEPT

-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT

Answer 1

-A POSTROUTING -o eth0 -j MASQUERADE

You asked it to masquerade traffic going out the internal interface, so that's what it's doing. Masquerading means to impersonate the source of the connection. Remove this line and, if something breaks, add the correct rule.

Answer 2

I FINALLY solved this.

Well, actually I gave up on it and found the answer while working on a different problem.

The answer is: The internal receiving system MUST use the same return route - same gateway / firewall system - for the port forwarding to work correctly! At least this is true for SMTP. It does not appear to be true for some services like ssh, but confirmed by behavior, SMTP (in my case Postfix) won't work correctly if the default route is other than the system that forwarded the connection.

This has implications for those of us who want redundancy - I'd like the mail server to be able to revcieve forwarded ports from more than one gateway / firewall. I have yet to figure out how to create special routes for this multiple gateway / firewall strategy, but I understand iptables can do it.

Good luck out there!