I'm looking for a way to distribute computing jobs (mainly rasterization work) across multiple, idle machines on the network. The servers are running Debian 6 right now.
Most of the work that needs to be done is standard command line work with some GhostScript rendering. Since rendering can take a very long time with just one CPU, I'd like to split up the work into many smaller pieces and have idle machines on the network help with the work.
Here is a rough workflow:
- User uploads a large PDF file to a web frontend
- The PDF is split into smaller pieces or moved to a SAN that can be accessed by all machines. All smaller pieces make up a "main task" that needs to be completed before the user can see any results.
- A job is scheduled for each page of the PDF file and put on a "queue".
- Idle machines grab a job from that queue and process/render the page of the PDF file.
- After completing, they place the resulting raster data onto the SAN and mark the job complete.
- Once all jobs for a "main task" have been completed, the user will see an overview of all rendered pages on the web frontend.
I'm aware about job schedulers, but since this is a new territory for me, I'm not even sure what search terms to use with Google.
So, I'd appreciate any pointers and suggestions. It would be ideal to have a de-centralized structure to avoid a single point of failure for all machines.