I'm looking for a recommendation on how to reliably run background operations. I have many servers posting operation messages to AWS SQS, and I have another server which will run these CPU intensive operations when idle.
I need some sort of polling mechanism to check SQS every so often for new operations. Is cron the best way to do this? I was thinking of putting a PHP script in cron and poll every 5 minutes - is this reliable?
I need to make sure one operation (message from SQS) is processed at a time. Some processing may take more than 5 minutes - how can I notify cron not to poll if operations are pending?
How can I run operations only when the CPU is idle? Does PHP make CPU data available?
Above all, I need this to be highly reliable. If processing on one operation fails, it should not affect future operations.
Can someone point me in the right direction?