Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm using celery 2.5.1 with django on a micro ec2 instance with 613mb memory and as such have to keep memory consumption down.

Currently I'm using it only for the scheduler "celery beat" as a web interface to cron, though I hope to use it for more in the future. I've noticed it is the biggest consumer of memory on my micro machine even though I have configured the number of workers to one. I don't have many other options set in settings.py:

import djcelery
djcelery.setup_loader()
BROKER_BACKEND = 'djkombu.transport.DatabaseTransport'
CELERYBEAT_SCHEDULER = 'djcelery.schedulers.DatabaseScheduler'
CELERY_RESULT_BACKEND = 'database'

BROKER_POOL_LIMIT = 2
CELERYD_CONCURRENCY = 1
CELERY_DISABLE_RATE_LIMITS = True
CELERYD_MAX_TASKS_PER_CHILD = 20
CELERYD_SOFT_TASK_TIME_LIMIT = 5 * 60
CELERYD_TASK_TIME_LIMIT = 6 * 60

Here's the details via top:

 PID USER  NI CPU%  VIRT   SHR  RES  MEM% Command 
1065 wuser 10  0.0  283M  4548  85m  14.3 python manage_prod.py celeryd --beat
1025 wuser 10  1.0  577M  6368  67m  11.2 python manage_prod.py celeryd --beat
1071 wuser 10  0.0  578M  2384  62m  10.6 python manage_prod.py celeryd --beat

That's about 214mb of memory (and not much shared) to run a cron job occasionally. Have I done anything wrong, or can this be reduced about ten-fold somehow? ;)

Update: here's my upstart config:

description "Celery Daemon"

start on (net-device-up and local-filesystems)
stop on runlevel [016]
nice 10
respawn
respawn limit 5 10
chdir /home/wuser/wuser/
env CELERYD_OPTS=--concurrency=1

exec sudo -u wuser -H /usr/bin/python manage_prod.py celeryd --beat --concurrency=1 --loglevel info --logfile /var/tmp/celeryd.log

Update 2:

I notice there is one root process, one user child process, and two grandchildren from that. So I think it isn't a matter of duplicate startup.

root  34580  1556 sudo -u wuser -H /usr/bin/python manage_prod.py celeryd 
wuser  577M 67548 └─ python manage_prod.py celeryd --beat --concurrency=1 
wuser  578M 63784    ├─  python manage_prod.py celeryd --beat --concurrency=1
wuser  271M 76260    └─  python manage_prod.py celeryd --beat --concurrency=1
share|improve this question
add comment

1 Answer

You can make sure that celery is only including the bare minimum of your code (I've seen celery configured to import people's entire web applications... not pretty). However, at the end of the day, you're looking at a very large chunk of Python, which is going to chew up a lot of memory by it's very nature.

If you want a low-memory task scheduling tool, I'd suggest real, honest-to-goodness cron.

share|improve this answer
    
+1 to this; celery may have started out as a scheduler, but its real strengths are in its worker model. Like @womble said, either stick to cron--or pick something that's specifically used for scheduling. –  Andrew M. Apr 7 '12 at 0:21
    
Thanks, you've given me some insight in to the problem. Unfortunately to do its tasks it needs to import a lot of the code, though I wouldn't expect it to be so large. In any case cron might not help much in absolute memory usage as it would have to run the same tasks ... however, (a big however) the cron job would release its memory when the task completed instead of sitting there in ram 98% of the time doing nothing. Maybe the "recycle" options can help. –  Gringo Suave Apr 7 '12 at 1:04
1  
Celery itself is a large quantity of code; removing that from the jobs you need to run would save you a fair amount of memory. –  womble Apr 7 '12 at 1:33
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.