A proper way to use AWS Redis Cluster with Celery

I have a Redis cluster in AWS, and I am trying to use that Redis as a broker with Celery. The error that I have is:

ValueError: Couldn't import '<module_name>': Port could not be cast to integer value as '<redis_host>:6379'

Here is the usage:

REDIS_ENDPOINT = config('REDIS_ENDPOINT')  # should be of the form 'hostname:port'
REDIS_HOST, REDIS_PORT = REDIS_ENDPOINT.split(':')
redis_url = f'redis://{REDIS_HOST}:{REDIS_PORT}'

app.conf.update(
    broker_url=redis_url,
    # worker_concurrency=1,
    worker_prefetch_multiplier=worker_prefetch_multiplier,
)

I really don't understand why it is not working. No need password btw. %100 sure the redis_host is in valid format because another Django app can connect easily with the same technic.

The working usage in Django backend:

REDIS_HOST, REDIS_PORT = REDIS_ENDPOINT.split(':')

CACHES = {
    'default': {
        'BACKEND': 'django_redis.cache.RedisCache',
        'LOCATION': f'redis://{REDIS_HOST}:{REDIS_PORT}/0',
        'OPTIONS': {
            'CLIENT_CLASS': 'django_redis.client.DefaultClient',
        },
        'KEY_PREFIX': 'my_cache',
        'TIMEOUT': CACHE_TTL,  # Use the CACHE_TTL setting for cache timeout
    }
}

Please do not forget to ping me ! Thanks for any help!

I also tried using redis-py-cluster lib in python:

REDIS_ENDPOINT = config('REDIS_ENDPOINT')  # should be of the form 'hostname:port'
REDIS_HOST, REDIS_PORT = REDIS_ENDPOINT.split(':')
redis_broker = Redis(host=REDIS_HOST, port=REDIS_PORT)
REDIS_NODES = [{"host": REDIS_HOST, "port": REDIS_PORT}]

app.conf.update(
    broker=redis_broker,
    # result_backend=result_backend,
    broker_url='redis://',
    broker_transport_options={
        'master_name': 'mymaster',
        'startup_nodes': REDIS_NODES
    },    # result_backend=result_backend,
    # worker_concurrency=1,
    worker_prefetch_multiplier=worker_prefetch_multiplier,
)

Not working.

I also used just redis lib. And also had the same error that I have when trying the first way.


REDIS_ENDPOINT = config('REDIS_ENDPOINT')  # should be of the form 'hostname:port'
REDIS_HOST, REDIS_PORT = REDIS_ENDPOINT.split(':')
result_backend = f'redis://{REDIS_HOST}:{REDIS_PORT}/0'
broker_url = f'redis://{REDIS_HOST}:{REDIS_PORT}/0'
redis_broker = Redis(host=REDIS_HOST, port=REDIS_PORT)

app.conf.update(
    broker=redis_broker,
    result_backend=result_backend,
    # result_backend=result_backend,
    # worker_concurrency=1,
    worker_prefetch_multiplier=worker_prefetch_multiplier,
)

Solution

I use Celery in dev and run a Redis Cluster on k8s (not AWS) so I'm interested in a solution as well for prod. The Celery team has been working on finding a solution since 2015.

According to a recent Celery Github discussion the Celery team doesn't have the funds currently to work on adding Redis Cluster support. A team member apparently made some progress but then delayed the remaining work to v6+?

There are several discussions on GH about other projects stalled on Redis Cluster support.

Here is a repo that claims to enable Celery backend support for connecting to a Redis Cluster. No mention of Celery broker support though.

After installing the celery-redis-cluster-backend package it uses the following for celeryconfig.py.

CELERY_RESULT_BACKEND = "celery_redis_cluster_backend.redis_cluster.RedisClusterBackend"
CELERY_REDIS_CLUSTER_SETTINGS = { 'startup_nodes': [
    {"host": "localhost", "port": "6379"},
    {"host": "localhost", "port": "6380"},
    {"host": "localhost", "port": "6381"}
]}