Questions tagged [celery]
Celery is a distributed task queue written in Python.
34
questions
1
vote
0
answers
118
views
Issue with DNS Resolution in Celery/Redis on Debian 12 - VPS Provider Specific
I'm facing a peculiar issue with DNS resolution in a Celery/Redis setup on some Debian 12 installations. It seems that the problem might also relate to the VPS network settings. I'm at a loss and ...
0
votes
1
answer
389
views
High CPU load on EC2 with Nginx/Celery/Django causes server to fail
I am running a Django web app on an EC2 server using Nginx, and uWSGI. I also have Celery running some background tasks (no CRON jobs, just on occasional user actions).
The app is in early closed Beta ...
4
votes
1
answer
7k
views
Memory limits in systemd
I am using systemd to run Celery, a python distributed task scheduler, on an Ubuntu 18.04 LTS server.
I have to schedule mostly long running tasks (taking several minutes to execute), which, depending ...
8
votes
0
answers
1k
views
How many celery and gunicorn workers on the same server?
I have one server that runs a Django application served by gunicorn and a Celery task queue.
Gunicorn docs suggest (2 x $num_cores) + 1 as the default number of workers.
Celery docs show that the ...
2
votes
0
answers
19
views
Celery: Finding the settings that we use
I have a Celery server that someone else set up, with a byzantine system to manage the configuration. I'd like to know the Cerely settings that our server uses, without looking at the code. I have an ...
1
vote
0
answers
405
views
Worker processes in Celery
I have a CPU intensive Celery task and within one of the task it can be further parallelized using joblib. By default, starting a worker with celery, will create a pool with number of max concurrency ...
1
vote
1
answer
1k
views
AWS autoscaling: How to kill the machines that are not busy
We have an autoscaling group that spawns worker servers. Each worker server runs celery processes. We monitor the celery queue length using cloudwatch and depending on that queue length we spawn/kill ...
3
votes
0
answers
196
views
Preventing Ubuntu EC2 server locking up with high CPU
I have a server that runs celery tasks. It runs a couple of worker threads with celery multi start 2, configured using systemd. Sometimes, it gets overworked and hits 100% CPU. When this happens, ...
0
votes
1
answer
2k
views
Remove consumer before start deployment from RabbitMQ queue
We have setup of Celery worker with 8 node setup. It create 8 queues in RabbitMQ.
When we start deploying new changes, last step in ansible playbook is Celery restart.
Celery restart has to down ...
1
vote
1
answer
423
views
ubuntu grant user permissions to run file without SU - running python celery
im having trouble with running celery as a daemon on ubuntu.
I have run the debug command as suggested in the documentation http://docs.celeryproject.org/en/latest/userguide/daemonizing.html#example-...
2
votes
2
answers
8k
views
What is the correct syntax to run cron every 48 hours?
Will this work correctly?
CELERYBEAT_SCHEDULE = {
'task-number-one': {
'task': 'project.users.tasks.send_something',
'schedule': crontab(minute='*/2880'),
}
}
1
vote
0
answers
515
views
Rabbitmq running out of memory due to "Processes / other"
I am running rabbitmq within kubernetes using the rabbitmq:3.6.11-management docker image (https://hub.docker.com/_/rabbitmq/). Looking at the management dashboard, there seems to be a lot of memory ...
1
vote
0
answers
1k
views
How to view pending messages in RabbitMQ admin?
How do you view pending/queued messages in RabbitMQ admin?
I have the RabbitMQ admin tool running, and the overview page is showing that there are over 4000 messages ready. It's also showing that ...
1
vote
1
answer
2k
views
Graceful shutdown of celery workers with systemd?
I have a box with celery workers, some of which perform reasonably long-running tasks (a few minutes). When this box updates, I'd like to shutdown the celery workers gracefully, so that ideally:
New ...
3
votes
1
answer
2k
views
rabbitmq: with hundreds of celery workers, beam.smp consumes > 200% CPU
I have one machine (test-server) with a rabbitmq server and 4 celery workers, and another machine (test-worker) with 240 celery workers, which connect to the rabbitmq server on test-server.
All ...
1
vote
1
answer
444
views
DatabaseError between Oracle and Django with Celery
I'm having trouble with what seems to be the very last step in the deployment of a Django-Celery-RabbitMQ project.
After running the celery, httpd and rabbitmq-server services through systemd, when I ...
0
votes
1
answer
1k
views
How to get logwatch to filter dates in celery logs?
I'm a newbie to Logwatch and have it set up to watch log files created by celery, a distributed task queue popular in the Python world.
My logwatch.conf "Range" is set to "between yesterday 4am and ...
3
votes
1
answer
1k
views
Docker, docker-compose application settings
I'm starting to migrate my Application to Docker containers:
I use Ngnix, supervisord, gunicorn, python flask, celery, flower, lighttpd, RabbitMQ and Postgresql.
In my original virtual machine, I keep ...
1
vote
1
answer
20k
views
How to stop celery worker?
I am running celery via redis. I start my worker like this:
celery multi start worker1 -A mypackage.tasks.tasks
celery multi v3.1.18 (Cipater)
> Starting nodes...
> worker1@polyphemus....
5
votes
1
answer
3k
views
Is there a way to use length of a RabbitMQ queue used by Celery to start instance in an autoscale group?
Is there any way for Celery to emit events when the length of a queue exceeds a threshold? I want to use that event to start an EC2-instance.
We have two queues for two different tasks in Celery. ...
1
vote
0
answers
5k
views
Supervisor start/stop/restart not behaving as expected
I am using supervisor to manage celery and due to occasional project requirements am required to add environment variables. I set these in the supervisor [environment] section of the .conf file but ...
3
votes
1
answer
8k
views
Is there a more convenient way to stop (or restart) a detached celery beat process?
Just to clarify things - I am working on a systemd service file celerybeat.service which should take care of the Celery beat. What I am doing at the moment is calling a script which reads /var/run/...
0
votes
0
answers
3k
views
Run shell script as Supervisord subprocess?
I have a Django 1.6.2 application that uses Celery 3.1.7 for asynchronous tasks. I'm starting my Celery workers using Supervisor. So far everything is working well except when I reboot my Debian 7.8 ...
0
votes
0
answers
270
views
VPN-tunnel renegotiation problems
I have an application running on top of Celery with redis as a broker. Redis is hosted at our datacenter but the workers are hosted on a separate line. We have setup two-way communcation between the ...
17
votes
3
answers
64k
views
Running multiple workers using Celery
I need to read from Rabbitmq and execute task in parallel using Celery in a single system.
[2014-12-30 15:54:22,374: INFO/Worker-1] ...
[2014-12-30 15:54:23,401: INFO/Worker-1] ...
[2014-12-30 15:...
1
vote
1
answer
2k
views
Supervisord as Windows Service on Cygwin
I am attempting to run Celery as a Windows Service using Supervisord. I followed the configuration laid out on the Celery site and here. I have set up a virtual environment to run supervisord through ...
2
votes
2
answers
3k
views
Celery with Upstart - processes are dying unexpectedly
When I'm running Celery with Upstart, after a while, the child processes or the main processes die without any trace.
The Upstart script I'm using (/etc/init/celery):
description "celery"
start on ...
0
votes
1
answer
387
views
celery tasks from one domain appear on another domain's django admin hosted on the same box
I have a rather funny problem that I didn't manage to find it elsewhere. I have 2 django-celery projects running with redis on 2 different domains, but apparently hosted on the same box. I don't have ...
1
vote
1
answer
2k
views
Celery and RabbitMQ - Tools for monitoring and callout alarms
I think we've reached the limit of usability of celery events and celery fllower.
Is there a good tool for monitoring status of Celery running under RabbitMQ?
In particular, I would like something ...
2
votes
1
answer
3k
views
Approach to auto scale celery servers based on broker(redis) queue size
I am working on a project which requires rolling out new celery servers if the broker(redis) queue is consistently higher than a predetermined threshold size and killing the new boxes when the queue ...
2
votes
2
answers
2k
views
kill -HUP is not working with celery daemon
So I have a shell script that daemonizes celery, and creates a bunch of workers running as a daemon. I would like to have a way to restart the celery task when the underlying source gets changed since ...
9
votes
1
answer
7k
views
Celery Daemon receives unregistered tasks
I installed Celery for my Django project following what the official tutorial / doc says. And it's working fine when I launch celery at the command line, I can see it receiving the tasks and execute ...
1
vote
0
answers
2k
views
OSError : [Errno 38] Function not implemented - Django Celery implementation
I installed django-celery and I tried to start up the worker server but I get an OSError that a function isn't implemented. I'm running CentOS release 5.4 (Final) on a VPS:
. broker -> amqp://...
6
votes
3
answers
5k
views
Is it possible to run automatically celery at startup?
I have python server based on django and celery. Each time computer restarts, apache2 starts so my server is working, BUT I have to restart celery manually (going to my project directory and executing ...