I'm daemonizing my Celery worker using Supervisord. The issue is I had a typo in my CELERY_BROKER_URL and the worker is not properly connecting to RabbitMQ.
When I run celery -A mysite report it shows the old environment variable.
My /etc/supervisor/conf.d/celery.conf file does not include the environment variables:
[program:celery]
command=/webapps/mysite/scripts/celery/celery_start
autostart=true
autorestart=true
user=myuser
stdout_logfile=/webapps/mysite/logs/celery.log
redirect_stderr = true
The environment variables are picked up via my virtual environment in the celery_start script:
#!/bin/sh
DJANGODIR=/webapps/mysite/mysite
# Activate the virtual environment.
cd $DJANGODIR
. /webapps/mysite/bin/activate
. /webapps/mysite/bin/postactivate
# Programs meant to be run under supervisor should not daemonize themselves
# (do not use --daemon).
exec celery -A mysite worker -E -l info --concurrency=2
When I check the CELERY_BROKER_URL environment variable after activating the environment it is correct. I've tried supervisorctl restart celery which doesn't pick up the new environment variable (celery -A mysite report shows the old CELERY_BROKER_URL). I've tried supervisorctl shutdown and then supervisord which also won't pick up the new environment variable.
When I run ps aux | grep 'celery worker' I don't see anything, presumably because Celery is daemonized by Supervisor, so I'm not sure of a way to completely destroy the current Celery process.
No matter what, it feels like Celery is not picking up the new environment variable. How could I make this happen?
[EDIT]
My Celery settings in settings.py are as follows:
# Celery settings.
CELERY_BROKER_URL = os.environ.get(
'BROKER_URL', 'amqp://guest:guest@127.0.0.1//')
CELERY_TASK_SOFT_TIME_LIMIT = 60
CELERY_RESULT_BACKEND = 'django-db'
And my mysite/celery.py file is:
from __future__ import absolute_import, unicode_literals
import os
from celery import Celery
# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'settings.local')
APP = Celery('mysite')
# Using a string here means the worker doesn't have to serialize
# the configuration object to child processes.
# - namespace='CELERY' means all celery-related configuration keys
# should have a `CELERY_` prefix.
APP.config_from_object('django.conf:settings', namespace='CELERY')
# Load task modules from all registered Django app configs.
APP.autodiscover_tasks()
@APP.task(bind=True)
def debug_task(self):
print('Request: {0!r}'.format(self.request))
It turns out I was using a password for my broker that for some reason contained an invalid character.
The password was #qahrKscbW#3!HkMJg#jFcyaOR7HtK%j08Jt$yY2.
What was happening was that my broker_url was invalid, and so it was defaulting back to ampq://guest:password instead of ampq://myuser:#qahrKscbW#3!HkMJg#jFcyaOR7HtK%j08Jt$yY2@localhost/mysite.
I changed the password to only use alphanumeric characters and it worked.