Reliably running background operations

Question

I'm looking for a recommendation on how to reliably run background operations. I have many servers posting operation messages to AWS SQS, and I have another server which will run these CPU intensive operations when idle.

1. I need some sort of polling mechanism to check SQS every so often for new operations. Is cron the best way to do this? I was thinking of putting a PHP script in cron and poll every 5 minutes - is this reliable?

2. I need to make sure one operation (message from SQS) is processed at a time. Some processing may take more than 5 minutes - how can I notify cron not to poll if operations are pending?

3. How can I run operations only when the CPU is idle? Does PHP make CPU data available?

Above all, I need this to be highly reliable. If processing on one operation fails, it should not affect future operations.

Can someone point me in the right direction?

Answer 1

I need some sort of polling mechanism to check SQS every so often for new operations. Is cron the best way to do this? I was thinking of putting a PHP script in cron and poll every 5 minutes - is this reliable?

That should be pretty reliable, unless you need something finer grained.

I need to make sure one operation (message from SQS) is processed at a time. Some processing may take more than 5 minutes - how can I notify cron not to poll if operations are pending?

Make your program in cron check for the existence of some stamp file. If it exists, then the invoked cron program simply exits without doing more work.

How can I run operations only when the CPU is idle? Does PHP make CPU data available?

If you run your CPU intensive operations via the scheduling class IDLE using chrt --idle command args

Note, this will only run through the processing queue when the processor has no other work to do, you may find that this behaviour leads to the process becoming far too slow! An off-shoot of this is that your load will always appear just under 1 (since the process will stay for long periods in the run queue without being scheduled).

Finally you can use the batch command to schedule tasks to execute when the load is below a certain value--frankly its not very dynamic and wont be held back once its started.