Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We have ~300 celeryd processes running under Ubuntu 10.4 64-bit , in idle every process takes ~19mb RES, ~174mb VIRT, thus - it's around 6GB of RAM in idle for all processes. In active state - process takes up to 100mb of RES and ~300mb VIRT

Every process uses minidom(xml files are < 500kb, simple structure) and urllib.

Quetions is - how can we decrease RAM consuption - at least for idle workers, probably some celery or python options may help? How to determine which part takes most of memory?

share|improve this question

1 Answer 1

Take a look at subclassing the AutoScaler class and setting the min_concurrency variable in __init__. The default min_concurrency of 0 is preventing the default AutoScaler to scale down.

I haven't tested this class (my Celery test nodes are shut down) but something like the following should work:

from celery.worker.autoscale import Autoscaler
class MinIdleAutoscaler(Autoscaler):
    def __init__(self,pool, max_concurrency, min_concurrency=10, keepalive=30, logger=None):
        Autoscaler.__init__(self,pool,max_concurrency,min_concurrency,keepalive,logger)

You can then tell Celery to use this class by setting CELERYD_AUTOSCALER in your Celery config.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.