0

I'm looking for a way to distribute computing jobs (mainly rasterization work) across multiple, idle machines on the network. The servers are running Debian 6 right now.

Most of the work that needs to be done is standard command line work with some GhostScript rendering. Since rendering can take a very long time with just one CPU, I'd like to split up the work into many smaller pieces and have idle machines on the network help with the work.

Here is a rough workflow:

  1. User uploads a large PDF file to a web frontend
  2. The PDF is split into smaller pieces or moved to a SAN that can be accessed by all machines. All smaller pieces make up a "main task" that needs to be completed before the user can see any results.
  3. A job is scheduled for each page of the PDF file and put on a "queue".
  4. Idle machines grab a job from that queue and process/render the page of the PDF file.
  5. After completing, they place the resulting raster data onto the SAN and mark the job complete.
  6. Once all jobs for a "main task" have been completed, the user will see an overview of all rendered pages on the web frontend.

I'm aware about job schedulers, but since this is a new territory for me, I'm not even sure what search terms to use with Google.

So, I'd appreciate any pointers and suggestions. It would be ideal to have a de-centralized structure to avoid a single point of failure for all machines.

1 Answer 1

1

I'm not sure if OpenSSI would help you, but it immediately came to my mind. It combines multiple servers as a single system image, which from administrator point of view is just a single server. Think it as an SMP server, but necessarily having multiple cores on one server, but on several servers, instead.

OpenSSI can spread the load among the cluster automatically and run the newly started processes on the least loaded node (or the fastest node). It can also automatically migrate the currently running processes from one node to another if it sees it beneficial. Of course, all this can also be done manually with command line utils.

If you don't want such kernel level approach to your problem or OpenSSI does not otherwise work for you, there's Torque which does this scheduling on application level.

1
  • Hi, thanks for the reply! I took a brief look at OpenSSI and I think the problem here is that we already have dozens of machines installed and each run a customized setup for a number of customers - so that might not work. TORQUE looks much more like what I have in mind at the first glance. I'll report back!
    – BastiBen
    Mar 27, 2012 at 8:20

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .