0

I have a Django 1.6.2 application that uses Celery 3.1.7 for asynchronous tasks. I'm starting my Celery workers using Supervisor. So far everything is working well except when I reboot my Debian 7.8 server. When that happens, my Celery workers won't restart because when the server reboots, it changes ownership of the celery log files from my "celery" user to "root". Also, the system deletes my /run/celery directory where I write my pid files. If I make these changes by hand and restart Celery, all my workers start normally.

Since these changes need to occur before starting the workers, I thought the solution would be to write a shell script which, due to its higher priority, gets executed from my supervisor.conf script before the celery worker commands (See below).

However, this setup script won't run. My supervisor log just says,

exited: celery-setup (exit status 0; not expected)
gave up: celery-setup entered FATAL state, too many start retries too quickly.

Also, no errors get written to the stdout/err logfiles.

My questions are:

  1. Is this the right approach to take to change my worker log permissions and recreate the pid directories before the celery workers are restarted?

  2. If this is the right approach, why isn't it working? If it's not, what's the right approach?

  3. If I were using an init.d celeryd daemon script instead of Supervisor, there's a CELERY_CREATE_DIRS setting that will automatically create pid and log directories that are owned by the user/group. Is there a way to replicate this setting when using supervisord?

On the one hand, I know Supervisor should only be used on foreground process, which is not the case with this script. On the other hand, I've seen other questions here which seem to imply that you should be able to run a shell script from Supervisor.

Thanks for your help.

# celery-supervisor-setup
#!/bin/bash

for i in 1 2 3
do
    if [ -f "/var/log/celery/worker${i}.log" ]; then
        echo "processing $i"
        chown celery:celery /var/log/celery/worker${i}.log
    fi
done

if [ ! -d "/run/celery" ]; then
    mkdir /run/celery
    chown celery:celery /run/celery
fi

# /etc/supervisor/conf.d/supervisor.conf
[program:celery-setup]
command = /www/myproj/conf/celery-supervisor-setup
; This next command didn't work
;command = bash -c " /www/myproj/conf/celery-supervisor-setup"
user = root
stdout_logfile = /var/log/celery_setup_stdout.log
stderr_logfile = /var/log/celery_setup_stderr.log
redirect_stderr = true
autostart = true
autorestart = false
priority=997

[program:celeryw1]
command=/home/myproj/venv/myproj/bin/celery worker --app=conf.celeryapp:app -n worker1 --config=celeryconfig -Q default --loglevel=info --pidfile=/var/run/celery/worker1.pid
directory=/www/myproj
user=celery
numprocs=1
stdout_logfile=/var/log/celery/worker1.log
stderr_logfile=/var/log/celery/worker1.log
redirect_stderr=true
autostart=true
autorestart=true
startsecs=1
stopwaitsecs=600
killasgroup=true
priority=998

; Note that I have "celeryw2" and "celeryw3" subprocess groups that are similar
; to the above except they refer to workers 2 and 3.  I omitted them to save space.

[group:celery-workers]
programs=celeryw1,celeryw2,celeryw3
priority=999
2
  • What about creating a wrapper script around celery, and run as root then su to celery in the wrapper script? The whole thing is pretty hacky, not sure why it's not working for you. Or you could try runit as an alternative to supervisor.
    – hookenz
    May 18, 2015 at 20:18
  • Thanks @Matt but that does sound too "hacky" to attempt. I've spent almost a month trying to get celery workers to start up via either an init.d daemon script or a supervisor script and I finally got it working with Supervisor so I'm really hesitant to muck with my Celery configuration or try switching to runit. I just don't understand why Debian is changing the ownership on these files and deleting my pid directory.
    – Jim
    May 18, 2015 at 20:33

0

You must log in to answer this question.

Browse other questions tagged .