Celery - minimize memory consuption

Question

We have ~300 celeryd processes running under Ubuntu 10.4 64-bit , in idle every process takes ~19mb RES, ~174mb VIRT, thus - it's around 6GB of RAM in idle for all processes. In active state - process takes up to 100mb of RES and ~300mb VIRT

Every process uses minidom(xml files are < 500kb, simple structure) and urllib.

Quetions is - how can we decrease RAM consuption - at least for idle workers, probably some celery or python options may help? How to determine which part takes most of memory?

Answer 1

Take a look at subclassing the AutoScaler class and setting the min_concurrency variable in __init__. The default min_concurrency of 0 is preventing the default AutoScaler to scale down.

I haven't tested this class (my Celery test nodes are shut down) but something like the following should work:

from celery.worker.autoscale import Autoscaler
class MinIdleAutoscaler(Autoscaler):
    def __init__(self,pool, max_concurrency, min_concurrency=10, keepalive=30, logger=None):
        Autoscaler.__init__(self,pool,max_concurrency,min_concurrency,keepalive,logger)

You can then tell Celery to use this class by setting CELERYD_AUTOSCALER in your Celery config.