Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

(Not sure if this is the right group)

I have a situation where I have a number of apps that get created/built. These apps change over time, so they need to be tested. I have a number of distributed test servers, so I can essentially test the apps in parallel on the different test boxes.

The situation I'm facing is one where we can have ~200-300 separate apps that need to be iteratively run/tested, where each run might take ~1-2 hours to run.

I'm looking for some kind of app that would allow me to create a batch/job kind of process, where each batch/job contains the app(s) that are to be run, and I can then schedule the batch to run as required. The process would then have the client machine request the batch/job to run when the machine is ready to run the next batch/job.

I've searched the usual Sourceforge/Freshmeat, as well as looked at some of the globus/cluster management kinds of apps. Haven't quite seen anything close to what I'm looking for.

Thoughts/comments would be helpful.


share|improve this question

Buildbot is a system to automate the compile/test cycle. Buildbot supports clusters of machines.

share|improve this answer
Buildbot is great. hudson-ci is popular as well, especially in the Java community. – Alex Holst Jan 29 '11 at 18:27

There are many answers to this question. Keep in mind however that all will require some scripting and customization on your part.

alvosu's suggestion of using Buildbot is excellent and you should definitely check that out first. Some other ideas I can think of include:

Distributed Hudson is worth investigating. Hudson is the standard continuous integration framework that everyone uses today, and you can distribute it's work among a farm of machines.

You mention you've already looked at cluster management software, that's another popular choice although more setup will be required. For example, an interesting approach would be to set up a ROCKS Cluster and use it's default remote execution engine SGE to launch jobs on build hosts.

I'm personally interested currently in the idea of using GNU Parallel as a simple queuing system.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.