Questions tagged [celery]

Celery is a distributed task queue written in Python.

Filter by
Sorted by
Tagged with
1 vote
0 answers
107 views

Issue with DNS Resolution in Celery/Redis on Debian 12 - VPS Provider Specific

I'm facing a peculiar issue with DNS resolution in a Celery/Redis setup on some Debian 12 installations. It seems that the problem might also relate to the VPS network settings. I'm at a loss and ...
Houman's user avatar
  • 1,635
0 votes
1 answer
387 views

High CPU load on EC2 with Nginx/Celery/Django causes server to fail

I am running a Django web app on an EC2 server using Nginx, and uWSGI. I also have Celery running some background tasks (no CRON jobs, just on occasional user actions). The app is in early closed Beta ...
artifex_knowledge's user avatar
4 votes
1 answer
7k views

Memory limits in systemd

I am using systemd to run Celery, a python distributed task scheduler, on an Ubuntu 18.04 LTS server. I have to schedule mostly long running tasks (taking several minutes to execute), which, depending ...
sblandin's user avatar
  • 141
7 votes
0 answers
1k views

How many celery and gunicorn workers on the same server?

I have one server that runs a Django application served by gunicorn and a Celery task queue. Gunicorn docs suggest (2 x $num_cores) + 1 as the default number of workers. Celery docs show that the ...
YPCrumble's user avatar
  • 165
2 votes
0 answers
19 views

Celery: Finding the settings that we use

I have a Celery server that someone else set up, with a byzantine system to manage the configuration. I'd like to know the Cerely settings that our server uses, without looking at the code. I have an ...
Ram Rachum's user avatar
  • 5,291
1 vote
0 answers
405 views

Worker processes in Celery

I have a CPU intensive Celery task and within one of the task it can be further parallelized using joblib. By default, starting a worker with celery, will create a pool with number of max concurrency ...
Coderji's user avatar
  • 69
1 vote
1 answer
1k views

AWS autoscaling: How to kill the machines that are not busy

We have an autoscaling group that spawns worker servers. Each worker server runs celery processes. We monitor the celery queue length using cloudwatch and depending on that queue length we spawn/kill ...
Ranjith Ramachandra's user avatar
3 votes
0 answers
192 views

Preventing Ubuntu EC2 server locking up with high CPU

I have a server that runs celery tasks. It runs a couple of worker threads with celery multi start 2, configured using systemd. Sometimes, it gets overworked and hits 100% CPU. When this happens, ...
user31415629's user avatar
0 votes
1 answer
2k views

Remove consumer before start deployment from RabbitMQ queue

We have setup of Celery worker with 8 node setup. It create 8 queues in RabbitMQ. When we start deploying new changes, last step in ansible playbook is Celery restart. Celery restart has to down ...
Nilesh's user avatar
  • 255
1 vote
1 answer
421 views

ubuntu grant user permissions to run file without SU - running python celery

im having trouble with running celery as a daemon on ubuntu. I have run the debug command as suggested in the documentation http://docs.celeryproject.org/en/latest/userguide/daemonizing.html#example-...
AlexW's user avatar
  • 13
1 vote
2 answers
8k views

What is the correct syntax to run cron every 48 hours?

Will this work correctly? CELERYBEAT_SCHEDULE = { 'task-number-one': { 'task': 'project.users.tasks.send_something', 'schedule': crontab(minute='*/2880'), } }
paolo's user avatar
  • 111
1 vote
0 answers
514 views

Rabbitmq running out of memory due to "Processes / other"

I am running rabbitmq within kubernetes using the rabbitmq:3.6.11-management docker image (https://hub.docker.com/_/rabbitmq/). Looking at the management dashboard, there seems to be a lot of memory ...
JBBreeman's user avatar
1 vote
0 answers
1k views

How to view pending messages in RabbitMQ admin?

How do you view pending/queued messages in RabbitMQ admin? I have the RabbitMQ admin tool running, and the overview page is showing that there are over 4000 messages ready. It's also showing that ...
Cerin's user avatar
  • 3,640
1 vote
1 answer
2k views

Graceful shutdown of celery workers with systemd?

I have a box with celery workers, some of which perform reasonably long-running tasks (a few minutes). When this box updates, I'd like to shutdown the celery workers gracefully, so that ideally: New ...
user31415629's user avatar
3 votes
1 answer
2k views

rabbitmq: with hundreds of celery workers, beam.smp consumes > 200% CPU

I have one machine (test-server) with a rabbitmq server and 4 celery workers, and another machine (test-worker) with 240 celery workers, which connect to the rabbitmq server on test-server. All ...
Isolated Ostrich's user avatar
1 vote
1 answer
443 views

DatabaseError between Oracle and Django with Celery

I'm having trouble with what seems to be the very last step in the deployment of a Django-Celery-RabbitMQ project. After running the celery, httpd and rabbitmq-server services through systemd, when I ...
vowel-house-might's user avatar
0 votes
1 answer
1k views

How to get logwatch to filter dates in celery logs?

I'm a newbie to Logwatch and have it set up to watch log files created by celery, a distributed task queue popular in the Python world. My logwatch.conf "Range" is set to "between yesterday 4am and ...
Ghopper21's user avatar
  • 113
3 votes
1 answer
1k views

Docker, docker-compose application settings

I'm starting to migrate my Application to Docker containers: I use Ngnix, supervisord, gunicorn, python flask, celery, flower, lighttpd, RabbitMQ and Postgresql. In my original virtual machine, I keep ...
gogasca's user avatar
  • 343
1 vote
1 answer
20k views

How to stop celery worker?

I am running celery via redis. I start my worker like this: celery multi start worker1 -A mypackage.tasks.tasks celery multi v3.1.18 (Cipater) > Starting nodes... > worker1@polyphemus....
blueFast's user avatar
  • 4,320
5 votes
1 answer
3k views

Is there a way to use length of a RabbitMQ queue used by Celery to start instance in an autoscale group?

Is there any way for Celery to emit events when the length of a queue exceeds a threshold? I want to use that event to start an EC2-instance. We have two queues for two different tasks in Celery. ...
web_ninja's user avatar
  • 153
1 vote
0 answers
5k views

Supervisor start/stop/restart not behaving as expected

I am using supervisor to manage celery and due to occasional project requirements am required to add environment variables. I set these in the supervisor [environment] section of the .conf file but ...
bgrantdev's user avatar
  • 111
3 votes
1 answer
8k views

Is there a more convenient way to stop (or restart) a detached celery beat process?

Just to clarify things - I am working on a systemd service file celerybeat.service which should take care of the Celery beat. What I am doing at the moment is calling a script which reads /var/run/...
DejanLekic's user avatar
0 votes
0 answers
3k views

Run shell script as Supervisord subprocess?

I have a Django 1.6.2 application that uses Celery 3.1.7 for asynchronous tasks. I'm starting my Celery workers using Supervisor. So far everything is working well except when I reboot my Debian 7.8 ...
Jim's user avatar
  • 340
0 votes
0 answers
270 views

VPN-tunnel renegotiation problems

I have an application running on top of Celery with redis as a broker. Redis is hosted at our datacenter but the workers are hosted on a separate line. We have setup two-way communcation between the ...
Bj Blazkowicz's user avatar
17 votes
3 answers
64k views

Running multiple workers using Celery

I need to read from Rabbitmq and execute task in parallel using Celery in a single system. [2014-12-30 15:54:22,374: INFO/Worker-1] ... [2014-12-30 15:54:23,401: INFO/Worker-1] ... [2014-12-30 15:...
SrC's user avatar
  • 181
1 vote
1 answer
2k views

Supervisord as Windows Service on Cygwin

I am attempting to run Celery as a Windows Service using Supervisord. I followed the configuration laid out on the Celery site and here. I have set up a virtual environment to run supervisord through ...
Jason Hawkins's user avatar
2 votes
2 answers
3k views

Celery with Upstart - processes are dying unexpectedly

When I'm running Celery with Upstart, after a while, the child processes or the main processes die without any trace. The Upstart script I'm using (/etc/init/celery): description "celery" start on ...
yprez's user avatar
  • 183
0 votes
1 answer
386 views

celery tasks from one domain appear on another domain's django admin hosted on the same box

I have a rather funny problem that I didn't manage to find it elsewhere. I have 2 django-celery projects running with redis on 2 different domains, but apparently hosted on the same box. I don't have ...
symbiotech's user avatar
1 vote
1 answer
2k views

Celery and RabbitMQ - Tools for monitoring and callout alarms

I think we've reached the limit of usability of celery events and celery fllower. Is there a good tool for monitoring status of Celery running under RabbitMQ? In particular, I would like something ...
Goro's user avatar
  • 674
2 votes
1 answer
3k views

Approach to auto scale celery servers based on broker(redis) queue size

I am working on a project which requires rolling out new celery servers if the broker(redis) queue is consistently higher than a predetermined threshold size and killing the new boxes when the queue ...
APZ's user avatar
  • 964
2 votes
2 answers
2k views

kill -HUP is not working with celery daemon

So I have a shell script that daemonizes celery, and creates a bunch of workers running as a daemon. I would like to have a way to restart the celery task when the underlying source gets changed since ...
Kevin Meyer's user avatar
9 votes
1 answer
7k views

Celery Daemon receives unregistered tasks

I installed Celery for my Django project following what the official tutorial / doc says. And it's working fine when I launch celery at the command line, I can see it receiving the tasks and execute ...
Bastian's user avatar
  • 283
1 vote
0 answers
1k views

OSError : [Errno 38] Function not implemented - Django Celery implementation

I installed django-celery and I tried to start up the worker server but I get an OSError that a function isn't implemented. I'm running CentOS release 5.4 (Final) on a VPS: . broker -> amqp://...
Jordan Messina's user avatar
6 votes
3 answers
5k views

Is it possible to run automatically celery at startup?

I have python server based on django and celery. Each time computer restarts, apache2 starts so my server is working, BUT I have to restart celery manually (going to my project directory and executing ...
user avatar