161 reputation
9
bio website
location
age
visits member for 4 years, 1 month
seen Dec 3 at 22:19

Happily getting paid to learn at Hacker School.

Used to work at an medical imaging lab.


Sep
24
awarded  Autobiographer
May
13
awarded  Self-Learner
Feb
11
awarded  Popular Question
Jan
29
awarded  Supporter
Apr
28
awarded  Scholar
Apr
28
accepted Parallel execution of different commands on multiple hosts
Apr
28
awarded  Critic
Apr
25
awarded  Teacher
Apr
25
answered Parallel execution of different commands on multiple hosts
Apr
25
comment Parallel execution of different commands on multiple hosts
Ah! It looks like the GNU version of parallel does do this - gnu.org/software/parallel/…. No load balancing, and I haven't tried it out to see what it does with each stdout, but I think this is precisely what I wanted. Unfortunately I've written a script that incorporates some load balancing, so I'll be sticking with it for now.
Apr
15
revised Parallel execution of different commands on multiple hosts
Update: I've got something working for my purposes, but it sucks.
Apr
14
revised Parallel execution of different commands on multiple hosts
Clarified even more
Apr
14
comment Parallel execution of different commands on multiple hosts
What I'd like to do doesn't require a merge (or split) step; I'd like to use this for tasks so embarrassingly parallel that they're already conveniently split for me, as in the gzipping 100 files example. I'm somewhat familiar with Condor, but as far as I can tell I'd need root access (on all machines) to install it properly.
Apr
14
comment Parallel execution of different commands on multiple hosts
I mean parallel as in parallel processing, or parallel computing - the different commands are most likely the same command run on a different portion of the whole dataset. If I'm gziping 100 files, I want to divide these 'commands' (there're all the same command, gzip, with different arguments) among different hosts. (assume a shared filesystem for this example)
Apr
14
comment Parallel execution of different commands on multiple hosts
Yeah, that bash script is pretty much what I want, but with some of the nice features of things like clusterssh. I couldn't find a lot of documentation for clusterssh; I only found references to it controlling many of its concept of terminal sessions - what else does it do? I only meant "just" to mean that like the other two tools mentioned so far, it doesn't do quite what I want.
Apr
14
revised Parallel execution of different commands on multiple hosts
added 72 characters in body
Apr
14
comment Parallel execution of different commands on multiple hosts
Again, it looks like clusterssh is just for running the same command on many hosts - I'd like to run different commands on each host.
Apr
13
revised Parallel execution of different commands on multiple hosts
wording
Apr
13
revised Parallel execution of different commands on multiple hosts
Made it more clear that I want to use DIFFERENT commands
Apr
13
comment Parallel execution of different commands on multiple hosts
As far as I can tell, both mussh and rshall just run the same command on each host - I'd like to run different commands on each host.