Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm using celery 2.5.1 with django on a micro ec2 instance with 613mb memory and as such have to keep memory consumption down.

Currently I'm using it only for the scheduler "celery beat" as a web interface to cron, though I hope to use it for more in the future. I've noticed it is the biggest consumer of memory on my micro machine even though I have configured the number of workers to one. I don't have many other options set in

import djcelery
BROKER_BACKEND = 'djkombu.transport.DatabaseTransport'
CELERYBEAT_SCHEDULER = 'djcelery.schedulers.DatabaseScheduler'


Here's the details via top:

1065 wuser 10  0.0  283M  4548  85m  14.3 python celeryd --beat
1025 wuser 10  1.0  577M  6368  67m  11.2 python celeryd --beat
1071 wuser 10  0.0  578M  2384  62m  10.6 python celeryd --beat

That's about 214mb of memory (and not much shared) to run a cron job occasionally. Have I done anything wrong, or can this be reduced about ten-fold somehow? ;)

Update: here's my upstart config:

description "Celery Daemon"

start on (net-device-up and local-filesystems)
stop on runlevel [016]
nice 10
respawn limit 5 10
chdir /home/wuser/wuser/
env CELERYD_OPTS=--concurrency=1

exec sudo -u wuser -H /usr/bin/python celeryd --beat --concurrency=1 --loglevel info --logfile /var/tmp/celeryd.log

Update 2:

I notice there is one root process, one user child process, and two grandchildren from that. So I think it isn't a matter of duplicate startup.

root  34580  1556 sudo -u wuser -H /usr/bin/python celeryd 
wuser  577M 67548 └─ python celeryd --beat --concurrency=1 
wuser  578M 63784    ├─  python celeryd --beat --concurrency=1
wuser  271M 76260    └─  python celeryd --beat --concurrency=1
share|improve this question

You can make sure that celery is only including the bare minimum of your code (I've seen celery configured to import people's entire web applications... not pretty). However, at the end of the day, you're looking at a very large chunk of Python, which is going to chew up a lot of memory by it's very nature.

If you want a low-memory task scheduling tool, I'd suggest real, honest-to-goodness cron.

share|improve this answer
+1 to this; celery may have started out as a scheduler, but its real strengths are in its worker model. Like @womble said, either stick to cron--or pick something that's specifically used for scheduling. – Andrew M. Apr 7 '12 at 0:21
Thanks, you've given me some insight in to the problem. Unfortunately to do its tasks it needs to import a lot of the code, though I wouldn't expect it to be so large. In any case cron might not help much in absolute memory usage as it would have to run the same tasks ... however, (a big however) the cron job would release its memory when the task completed instead of sitting there in ram 98% of the time doing nothing. Maybe the "recycle" options can help. – Gringo Suave Apr 7 '12 at 1:04
Celery itself is a large quantity of code; removing that from the jobs you need to run would save you a fair amount of memory. – womble Apr 7 '12 at 1:33

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.