Is there any way for Celery to emit events when the length of a queue exceeds a threshold? I want to use that event to start an EC2-instance.
We have two queues for two different tasks in Celery. One of these queues has tasks which will require an instance to be started in the autoscaling group. In the first autoscaling group which handles one kind of tasks, we always have one instance running. It is then easy to start another instance using Amazon CloudWatch. But for the other autoscaling group we don't want it to have any instances if there are no tasks because many a time there might not be tasks to be performed by this group. So only if the length of the Celery queue for this task goes above a given threshold, we want to start an instance in this autoscaling group. What would be a good way to do this?