Questions tagged [celery]

Celery is a distributed task queue written in Python.

Filter by
Sorted by
Tagged with
0
votes
0answers
53 views

How to add queues to rabbitmq + redis + celery?

I am new to backend development and I will try to explain my situation: Stack: Flask (nginx, uwsgi) Celery Redis (BackEnd) RabbitMQ (Broker) MongoDB (stores a lot of data) What do I want to ...
0
votes
1answer
104 views

High CPU load on EC2 with Nginx/Celery/Django causes server to fail

I am running a Django web app on an EC2 server using Nginx, and uWSGI. I also have Celery running some background tasks (no CRON jobs, just on occasional user actions). The app is in early closed Beta ...
1
vote
1answer
940 views

Memory limits in systemd

I am using systemd to run Celery, a python distributed task scheduler, on an Ubuntu 18.04 LTS server. I have to schedule mostly long running tasks (taking several minutes to execute), which, depending ...
3
votes
0answers
533 views

How many celery and gunicorn workers on the same server?

I have one server that runs a Django application served by gunicorn and a Celery task queue. Gunicorn docs suggest (2 x $num_cores) + 1 as the default number of workers. Celery docs show that the ...
2
votes
0answers
9 views

Celery: Finding the settings that we use

I have a Celery server that someone else set up, with a byzantine system to manage the configuration. I'd like to know the Cerely settings that our server uses, without looking at the code. I have an ...
1
vote
0answers
137 views

Worker processes in Celery

I have a CPU intensive Celery task and within one of the task it can be further parallelized using joblib. By default, starting a worker with celery, will create a pool with number of max concurrency ...
1
vote
1answer
608 views

AWS autoscaling: How to kill the machines that are not busy

We have an autoscaling group that spawns worker servers. Each worker server runs celery processes. We monitor the celery queue length using cloudwatch and depending on that queue length we spawn/kill ...
3
votes
0answers
120 views

Preventing Ubuntu EC2 server locking up with high CPU

I have a server that runs celery tasks. It runs a couple of worker threads with celery multi start 2, configured using systemd. Sometimes, it gets overworked and hits 100% CPU. When this happens, ...
0
votes
1answer
1k views

Remove consumer before start deployment from RabbitMQ queue

We have setup of Celery worker with 8 node setup. It create 8 queues in RabbitMQ. When we start deploying new changes, last step in ansible playbook is Celery restart. Celery restart has to down ...
1
vote
1answer
299 views

ubuntu grant user permissions to run file without SU - running python celery

im having trouble with running celery as a daemon on ubuntu. I have run the debug command as suggested in the documentation http://docs.celeryproject.org/en/latest/userguide/daemonizing.html#example-...
0
votes
2answers
4k views

What is the correct syntax to run cron every 48 hours?

Will this work correctly? CELERYBEAT_SCHEDULE = { 'task-number-one': { 'task': 'project.users.tasks.send_something', 'schedule': crontab(minute='*/2880'), } }
1
vote
0answers
393 views

Rabbitmq running out of memory due to "Processes / other"

I am running rabbitmq within kubernetes using the rabbitmq:3.6.11-management docker image (https://hub.docker.com/_/rabbitmq/). Looking at the management dashboard, there seems to be a lot of memory ...
1
vote
0answers
1k views

How to view pending messages in RabbitMQ admin?

How do you view pending/queued messages in RabbitMQ admin? I have the RabbitMQ admin tool running, and the overview page is showing that there are over 4000 messages ready. It's also showing that ...
0
votes
1answer
2k views

Graceful shutdown of celery workers with systemd?

I have a box with celery workers, some of which perform reasonably long-running tasks (a few minutes). When this box updates, I'd like to shutdown the celery workers gracefully, so that ideally: New ...
3
votes
1answer
2k views

rabbitmq: with hundreds of celery workers, beam.smp consumes > 200% CPU

I have one machine (test-server) with a rabbitmq server and 4 celery workers, and another machine (test-worker) with 240 celery workers, which connect to the rabbitmq server on test-server. All ...
1
vote
1answer
325 views

DatabaseError between Oracle and Django with Celery

I'm having trouble with what seems to be the very last step in the deployment of a Django-Celery-RabbitMQ project. After running the celery, httpd and rabbitmq-server services through systemd, when I ...
0
votes
1answer
798 views

How to get logwatch to filter dates in celery logs?

I'm a newbie to Logwatch and have it set up to watch log files created by celery, a distributed task queue popular in the Python world. My logwatch.conf "Range" is set to "between yesterday 4am and ...
3
votes
1answer
1k views

Docker, docker-compose application settings

I'm starting to migrate my Application to Docker containers: I use Ngnix, supervisord, gunicorn, python flask, celery, flower, lighttpd, RabbitMQ and Postgresql. In my original virtual machine, I keep ...
1
vote
1answer
15k views

How to stop celery worker?

I am running celery via redis. I start my worker like this: celery multi start worker1 -A mypackage.tasks.tasks celery multi v3.1.18 (Cipater) > Starting nodes... > worker1@polyphemus....
5
votes
1answer
3k views

Is there a way to use length of a RabbitMQ queue used by Celery to start instance in an autoscale group?

Is there any way for Celery to emit events when the length of a queue exceeds a threshold? I want to use that event to start an EC2-instance. We have two queues for two different tasks in Celery. ...
1
vote
0answers
4k views

Supervisor start/stop/restart not behaving as expected

I am using supervisor to manage celery and due to occasional project requirements am required to add environment variables. I set these in the supervisor [environment] section of the .conf file but ...
2
votes
1answer
6k views

Is there a more convenient way to stop (or restart) a detached celery beat process?

Just to clarify things - I am working on a systemd service file celerybeat.service which should take care of the Celery beat. What I am doing at the moment is calling a script which reads /var/run/...
0
votes
0answers
3k views

Run shell script as Supervisord subprocess?

I have a Django 1.6.2 application that uses Celery 3.1.7 for asynchronous tasks. I'm starting my Celery workers using Supervisor. So far everything is working well except when I reboot my Debian 7.8 ...
0
votes
0answers
220 views

VPN-tunnel renegotiation problems

I have an application running on top of Celery with redis as a broker. Redis is hosted at our datacenter but the workers are hosted on a separate line. We have setup two-way communcation between the ...
16
votes
3answers
46k views

Running multiple workers using Celery

I need to read from Rabbitmq and execute task in parallel using Celery in a single system. [2014-12-30 15:54:22,374: INFO/Worker-1] ... [2014-12-30 15:54:23,401: INFO/Worker-1] ... [2014-12-30 15:...
1
vote
1answer
2k views

Supervisord as Windows Service on Cygwin

I am attempting to run Celery as a Windows Service using Supervisord. I followed the configuration laid out on the Celery site and here. I have set up a virtual environment to run supervisord through ...
2
votes
2answers
3k views

Celery with Upstart - processes are dying unexpectedly

When I'm running Celery with Upstart, after a while, the child processes or the main processes die without any trace. The Upstart script I'm using (/etc/init/celery): description "celery" start on ...
0
votes
1answer
337 views

celery tasks from one domain appear on another domain's django admin hosted on the same box

I have a rather funny problem that I didn't manage to find it elsewhere. I have 2 django-celery projects running with redis on 2 different domains, but apparently hosted on the same box. I don't have ...
1
vote
1answer
2k views

Celery and RabbitMQ - Tools for monitoring and callout alarms

I think we've reached the limit of usability of celery events and celery fllower. Is there a good tool for monitoring status of Celery running under RabbitMQ? In particular, I would like something ...
2
votes
1answer
3k views

Approach to auto scale celery servers based on broker(redis) queue size

I am working on a project which requires rolling out new celery servers if the broker(redis) queue is consistently higher than a predetermined threshold size and killing the new boxes when the queue ...
2
votes
2answers
2k views

kill -HUP is not working with celery daemon

So I have a shell script that daemonizes celery, and creates a bunch of workers running as a daemon. I would like to have a way to restart the celery task when the underlying source gets changed since ...
9
votes
1answer
7k views

Celery Daemon receives unregistered tasks

I installed Celery for my Django project following what the official tutorial / doc says. And it's working fine when I launch celery at the command line, I can see it receiving the tasks and execute ...
1
vote
0answers
1k views

OSError : [Errno 38] Function not implemented - Django Celery implementation

I installed django-celery and I tried to start up the worker server but I get an OSError that a function isn't implemented. I'm running CentOS release 5.4 (Final) on a VPS: . broker -> amqp://...
6
votes
3answers
4k views

Is it possible to run automatically celery at startup?

I have python server based on django and celery. Each time computer restarts, apache2 starts so my server is working, BUT I have to restart celery manually (going to my project directory and executing ...