I recently wrote some scientific software designed to be run from the command line, and am now working on creating a web front end for that software to make it more accessible to scientists who aren't as savvy with the terminal. Implementing the front end is going to be trivial, but on the back end I want to make sure that I don't have too many requests executed at once (which would bog down the server). I had this issue a few years ago (when I had much less experience), and I simply hacked together a solution based on Perl scripts, MySQL, and crontab. I need similar functionality for this new system, but I have a hard time believing there isn't at least one mature open source solution for this.

First, is load management even the correct term for this? (if not, feel free to edit the title) Second, are there any open source solutions for what I'm trying to do?


  • What would you do with the "too many" requests: discard them, queue them, or ..? Also, what is the protocol between your web front-end and your software: http or something else? – Khaled Feb 22 '12 at 13:09
  • @Khaled In the case of too many simultaneous requests, the latter ones would be placed in a queue to be executed when more resources are available. – Daniel Standage Feb 22 '12 at 14:03
  • Maybe you're over complicating things, have a look into a message queuing system like RabbitMQ, Beanstalkd or Gearmand and see if anything like that would be suitable. Just put every request in the queue, if there isn't a queue for a particular function/node/whatever it will be run almost instantly, otherwise it will sit there till it's ready to be dealt with. – Smudge Feb 22 '12 at 14:31

What I think you are after is a resource management system like SGE (Sun Grid Engine, now Orcale Grid Engine but there are good forks).

SGE is typically used on a computer farm or high-performance computing (HPC) cluster and is responsible for accepting, scheduling, dispatching, and managing the remote and distributed execution of large numbers of standalone, parallel or interactive user jobs. It also manages and schedules the allocation of distributed resources such as processors, memory, disk space,and software licenses.

SGE (forks):

Also have a look at SLURM:

  • I took a quick look at both of these, but they both seem to be designed for distributed computing. My system will involve a single server running multiple simultaneous processes. Maybe these are will still satisfy my needs, but I just wanted to make sure what I was asking for is clear. – Daniel Standage Feb 22 '12 at 14:07
  • They will satisfy your needs, no big difference between one or X machines, it's still resource management.. – rkthkr Feb 27 '12 at 8:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.