3

Is there any way for Celery to emit events when the length of a queue exceeds a threshold? I want to use that event to start an EC2-instance.

We have two queues for two different tasks in Celery. One of these queues has tasks which will require an instance to be started in the autoscaling group. In the first autoscaling group which handles one kind of tasks, we always have one instance running. It is then easy to start another instance using Amazon CloudWatch. But for the other autoscaling group we don't want it to have any instances if there are no tasks because many a time there might not be tasks to be performed by this group. So only if the length of the Celery queue for this task goes above a given threshold, we want to start an instance in this autoscaling group. What would be a good way to do this?

6

I'd suggest a cron job to publish the current queue length as a CloudWatch custom metric, which will allow you to create auto-scaling alarms based on its value.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.