131 reputation
3
bio website insignificancegalore.net
location
age 40
visits member for 2 years, 6 months
seen Sep 25 at 6:46

Sep
24
awarded  Teacher
Nov
27
answered How to set ExecutionPolicy: access to registry key denied
Aug
8
awarded  Supporter
May
22
comment Lightweight distributed job queuing system for independent realtime jobs
Yes, the CPU is the limiting factor in the sense that each job maxes a single core. In our current setup, we simply run the jobs 12 at a time on a 12-core CPU and let the process scheduler take care of things. What I am looking for is a simple way to bring more servers into the mix.
May
22
comment Lightweight distributed job queuing system for independent realtime jobs
Edited the question to (hopefully) clarify this. Also: our goal is very modest: to take the turnaround time down from days (jobs run serially on one CPU) to hours (jobs run in parallel on multiple machines). I agree that it should not be complicated to set up our own system for this -- but I would rather just use an existing system.
May
22
awarded  Editor
May
22
revised Lightweight distributed job queuing system for independent realtime jobs
Clarified that the jobs are atomic, and that the single job runtime cannot be reduced.
May
22
comment Lightweight distributed job queuing system for independent realtime jobs
Yes, the jobs are totally independent -- but also atomic: These are heavy structural optimization calculations, and the single-job run time is about as low as we expect to get it (for the current CPU frequencies, at least). With respect to the farm of cheap systems: this is exactly what we are aiming for -- but I figured that the problem is so generic that maybe an good scheduler for this special case already existed.
May
22
asked Lightweight distributed job queuing system for independent realtime jobs