Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Possible Duplicate:
Linux - Running The Same Command on Many Machines at Once

Here's the situation:

  • We have a lab that consists of fifteen quad-core machines, each running Ubuntu Linux.
  • There is a video encoding software I need to be running, but one job (i.e. one video with one configuration) takes a really long time (couple of hours).
  • There are about as many configurations as machines (about 15) and around 20 videos.

So I thought about having the videos accessible through a centralized storage, but let each machine run the encoding process.

In its most basic form, the command used is something like this

./encode -d default.conf -f local.conf -i inputFile.yuv

Now, the question is: Is there any software that I could use to easily deploy those tasks on the lab machines? I was thinking about:

  • Having one master that dispatches jobs, e.g. "Tell machine 1 to run /home/user/encode -i input1.yuv, then /home/user/encode -i input2.yuv, et cetera"
  • Being able to see which node is currently working on which task and for how long
  • Being able to stop a task or retry upon failure

I am not limited to CLI, could also be a GUI application. Any ideas?

share|improve this question

marked as duplicate by Caleb, Smudge, Iain, Scott Pack, womble Aug 2 '11 at 11:54

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

I'm all for closing this myself -- some very good suggestions in there. Also related: What is a good modern parallel SSH tool? – slhck Aug 1 '11 at 9:55

Consider installing TORQUE. Its scheduler isn't the best out there, but it's more than sufficient for this kind of usage. You can replace the scheduler with Maui if you need the extra features later.

The only feature from your list TORQUE misses is automatically retrying a job on failure. But you should be able to script that yourself on the TORQUE server by checking the output of its qstat command to know which jobs are running, and the contents of your output folder to know what's finished.

share|improve this answer

And if you need a bigger solution with more options and features consider the Sun Grid Engine (SGE) now known as the Oracle Grid Engine.

share|improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.