Is it possible to have multiple remote nodes behind a NAT with GNU parallel?

Suppose some of a GNU parallel cluster exists behind a NAT (which may or may not be accessible only via a single IPv4 address through an ISP operating only IPv4) relative to the master node. ie more than one PC node having its own subnet ip address existing on some subnet other than that which hosts the master node.

Is there a way for GNU-parallel to distribute work through the NAT to all of those nodes?

After some research and thinking about the issue including a look at this somewhat related question

The only means I can think of would be to manually define a different port for each node using the -p flag described in the manual and then manually add a port forward rule in the NAT for each node:port

Is there some "trick" in gnu-parallel where jobs could be passed to one node behind the NAT then handed out from there to other nodes in its subnet?

Or perhaps there is a method by which slave nodes could pass a message via an https POST with a cron job and somehow generate and hold an established connection over a public port? Similar to how one can obtain public keys as described in this question (this idea is definietly outside my understanding of TCP/IPTABLES so I do realise that it may be fundamentally flawed on its face)

A solution which could be implimented solely within the master node and slave nodes would be preferable over one in which NAT entries were required.

  • The obvious solution is IPv6. – Michael Hampton Sep 27 '15 at 23:55
  • @Michael Hampton - Added note to clarify. – Mr Purple Sep 28 '15 at 4:41

There are several solutions.

The workers are behind NAT. No access to a jump host. No access to firewall.

For this you need some sort of VPN that can traverse the firewall. A TOR hidden service for port 22 on the workers can be used here. If all the workers are TOR-enabled:

parallel --ssh 'torsocks ssh' -S zij4uclus7xhwlhz.onion,isj4uclus7xhwlhz.onion,lzw4uclus7xhwlhz.onion echo ::: 1

If only some are:

parallel -S 'torsocks ssh zij4uclus7xhwlhz.onion,torsocks ssh isj4uclus7xhwlhz.onion,torsocks ssh lzw4uclus7xhwlhz.onion' echo ::: 1

The workers are behind NAT. No access to a jump host. Access to firewall.

If you can forward ports, so that port 2001 is port 22 on host 1, port 2002 is port 22 on host 2, 2003 is host 3 ... then you can use -p:

parallel -S 'ssh -p 2001 firewall,ssh -p 2002 firewall,ssh -p 2003 firewall' echo ::: 1

You can put this into .ssh/config:

Host host1.v
  Port 2001
Host host2.v
  Port 2002
Host host3.v
  Port 2003

Host *.v
  Hostname firewall

And then simply use host[1-3].v as normal hosts:

parallel -S host1.v,host2.v,host3.v echo ::: 1

The workers are behind NAT. Access to a jump host.

If you have access to a jump host from which you can reach the workers the obvious would be:

parallel --ssh 'ssh jumphost ssh' -S host1 echo ::: DOES NOT WORK

But this does not work because the command is dequoted by ssh twice where as GNU Parallel only expects it to be dequoted once.

So instead use .ssh/config again:

Host host1 host2 host3
  ProxyCommand ssh jump.host.domain nc -w 1 %h 22

It requires nc(netcat) to be installed on jumphost.

  • oooh Jump host sounds perfect! presumably If one has enough access to turn one node into a slave with ssh then one has enough access to make that the jump host too. So my naive 'remote master node' just needs to be a jumphost/have netcat and I need to make some changes to the local sshlogin file. Gold! Using tor would be nice solution for those without any access to port foward though - amazing! – Mr Purple Sep 28 '15 at 8:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.