In short, I have a cluster system at my hands, for which I have to route traffic through its master. Routing traffic from the nodes to the outside world works, but routing traffic from our department's subnet to the nodes fails. Unfortunately, adding the nodes to our subnet is out of the question.
The cluster consists of a master and a number of nodes, along with some periphery. The nodes are on an internal network, hidden from our intranet or the internet. A NAT is already in place on the master, so the nodes have access to internal and external servers. This part works.
The external interface of the master is on the same subnet as our workstation pcs, which share a gateway outside of our control.
Edit: The cluster runs CentOS 7, the pcs run a Ubuntu xenial-based distribution.
Some of our software packages need direct access to the nodes. For this, we wanted to set up second NAT on the master using iptables, and add an ip route on the pcs to send traffic to 10.10.1.0/24 through master.
master: ip route
default via 184.108.40.206 dev eth0 proto static metric 100 10.10.0.0/16 dev eth1 proto kernel scope link src 10.10.0.1 220.127.116.11/23 dev eth0 proto kernel scope link src 18.104.22.168 metric 100
master: iptables -vnL -t nat
Chain PREROUTING (policy ACCEPT 7356 packets, 880K bytes) pkts bytes target prot opt in out source destination Chain INPUT (policy ACCEPT 4884 packets, 687K bytes) pkts bytes target prot opt in out source destination Chain OUTPUT (policy ACCEPT 3445 packets, 225K bytes) pkts bytes target prot opt in out source destination Chain POSTROUTING (policy ACCEPT 3445 packets, 225K bytes) pkts bytes target prot opt in out source destination 439 33324 MASQUERADE all -- * eth0 10.10.1.0/24 0.0.0.0/0 61828 3710K MASQUERADE all -- * eth1 22.214.171.124/23 10.10.1.0/24
Using SNAT instead of MASQUERADE makes no difference.
node: ip route
default via 10.10.0.1 dev eth1 10.10.0.0/16 dev eth1 proto kernel scope link src 10.10.1.1
pc: ip route
default via 126.96.36.199 dev eth0 proto static metric 100 10.10.0.0/16 via 188.8.131.52 dev eth0 184.108.40.206/23 dev eth0 proto kernel scope link src 220.127.116.11 metric 100
Diagnosis so far
- NAT from node01 to internet/intranet/pcs works flawlessly.
- NAT from pc1 to node01 fails during TCP handshake:
- SYN is passed through master to node01, is marked as SYN_RECV in tcpdump
- SYN+ACK is sent from node01 to master
- SYN+ACK appears in tcpdump on master, is passed to filter
- tcpdump shows the SYN+ACK passing through to filter
- iptables shows the SYN+ACK packages passing through filter FORWARD, mangle FORWARD + POSTROUTING
- The SYN+ACK packages never make it through nat POSTROUTING (should they?)
- The SYN+ACK packages never arrive at pc1
- Of course, the handshake fails
- pc1 is stuck in SYN_SENT
- node01 is stuck in SYN_RECV
- eventually, the connection times out
- I have no way of monitoring packages at the gateway
My best guess is that a stateful router on the way drops the SYN+ACK package due to its source address being rewritten on the master, so its relation to the original SYN package is lost.
How can we get this working?
Let me know if additional configs/logs are needed.