I have a few terabytes of data that are processed by a tool. The output of the tool should be sent to two other machines simultaneously (simultaneously, because there's downtime while the tool runs and I want to limit the downtime).

If I just pipe from one machine then it's easy:

tool terabyte.txt | ssh user@ /sbin/process-input

but how can I send the data to multiple machines simultaneously?

I do not mind if the situation involves other software or scripts. it doesn't have to be a "pure ssh" solution.


You can try with ClutterSSH :

With ClusterSSH it is possible to make a SSH connection to multiple servers or machines to perform tasks from one single command window, without any scripting. The ‘cssh’ command lets you connect to any server specified as a command line argument, or to groups of servers (or cluster nodes) defined in a configuration file.


You could try using tee

tool terabyte.txt | tee >(ssh user@ /sbin/process-input) | ssh user@ /sbin/process-input

Perhaps, pee from moreutils might be even better:

pee is like tee but for pipes. Each command is run and fed a copy of the standard input. The output of all commands is sent to stdout.

So,you can run

tool terabyte.txt |pee 'ssh user@ /sbin/process-input' 'ssh user@ /sbin/process-input'


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.