1

In short, I have a cluster system at my hands, for which I have to route traffic through its master. Routing traffic from the nodes to the outside world works, but routing traffic from our department's subnet to the nodes fails. Unfortunately, adding the nodes to our subnet is out of the question.

The Setup

cluster setup, without dumb switches for simplicity

The cluster consists of a master and a number of nodes, along with some periphery. The nodes are on an internal network, hidden from our intranet or the internet. A NAT is already in place on the master, so the nodes have access to internal and external servers. This part works.

The external interface of the master is on the same subnet as our workstation pcs, which share a gateway outside of our control.

Edit: The cluster runs CentOS 7, the pcs run a Ubuntu xenial-based distribution.

The Task

Some of our software packages need direct access to the nodes. For this, we wanted to set up second NAT on the master using iptables, and add an ip route on the pcs to send traffic to 10.10.1.0/24 through master.

The Configuration

master: ip route

default via 123.45.67.254 dev eth0 proto static metric 100
10.10.0.0/16 dev eth1 proto kernel scope link src 10.10.0.1
123.45.67.0/23 dev eth0 proto kernel scope link src 123.45.67.204 metric 100

master: iptables -vnL -t nat

Chain PREROUTING (policy ACCEPT 7356 packets, 880K bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain INPUT (policy ACCEPT 4884 packets, 687K bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 3445 packets, 225K bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain POSTROUTING (policy ACCEPT 3445 packets, 225K bytes)
 pkts bytes target     prot opt in     out     source               destination
  439 33324 MASQUERADE  all  --  *     eth0    10.10.1.0/24         0.0.0.0/0
61828 3710K MASQUERADE  all  --  *     eth1    123.45.67.0/23       10.10.1.0/24

Using SNAT instead of MASQUERADE makes no difference.

node: ip route

default via 10.10.0.1 dev eth1
10.10.0.0/16 dev eth1 proto kernel scope link src 10.10.1.1

pc: ip route

default via 123.45.67.254 dev eth0  proto static  metric 100
10.10.0.0/16 via 123.45.67.204 dev eth0
123.45.67.0/23 dev eth0  proto kernel  scope link  src 123.45.67.191  metric 100

Diagnosis so far

  • NAT from node01 to internet/intranet/pcs works flawlessly.
  • NAT from pc1 to node01 fails during TCP handshake:
    • SYN is passed through master to node01, is marked as SYN_RECV in tcpdump
    • SYN+ACK is sent from node01 to master
    • SYN+ACK appears in tcpdump on master, is passed to filter
    • tcpdump shows the SYN+ACK passing through to filter
    • iptables shows the SYN+ACK packages passing through filter FORWARD, mangle FORWARD + POSTROUTING
    • The SYN+ACK packages never make it through nat POSTROUTING (should they?)
    • The SYN+ACK packages never arrive at pc1
  • Of course, the handshake fails
    • pc1 is stuck in SYN_SENT
    • node01 is stuck in SYN_RECV
    • eventually, the connection times out
  • I have no way of monitoring packages at the gateway

My best guess is that a stateful router on the way drops the SYN+ACK package due to its source address being rewritten on the master, so its relation to the original SYN package is lost.

How can we get this working?

Let me know if additional configs/logs are needed.

  • Come to think of it, the SYN+ACK packages visible in iptables might only be the SYN packages from pc1. – Erik E. Lorenz Feb 16 '18 at 8:29
  • 1
    Return packets should not go through nat POSTROUTING, they should be handled by the connection tracking. Add a match on ! --destination 10.10.1.0/24. Edit: I'm wondering if the masquerade on source 123.45.67.0/23 should be done. – wurtel Feb 16 '18 at 12:06
  • Masquerade isn't required. The whole eth1 entry can safely be removed since ACCEPT is our default. The return packages still don't make it to the pc, although they show up in tcpdump on the master. – Erik E. Lorenz Feb 19 '18 at 10:34
  • ! --destination 10.10.1.0/24 on the eth1 rule is obsolete now. Any commands to view all outgoing packets after iptables? tcpdump is before the filter step, right? – Erik E. Lorenz Feb 19 '18 at 10:44
  • The return packet seem to be dropped by the gateway. Connecting one of the pcs to the same switch via the VLAN of the public interface leads to the desired result. Connect it to the ethernet port in the same room (one company switch inbetween), in the same VLAN, with the same exact IP address, and the packets are dropped again. – Erik E. Lorenz Feb 19 '18 at 10:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.