[SOLVED] Django + Celery + SQS - Messages received but not executed by celery worker?

Django + Celery + SQS - Messages received but not executed by celery worker?

I'm trying to implement background tasks on my Django web app using Celery and AWS SQS as the message broker. I have a view which calls a task, which is sent to SQS successfully. In the console window where I have started the Celery worker, I get a statement saying the the task was received. However, the actual task never gets executed (i.e. the print statements I have added in the task never get printed).

I have configured Celery in my settings.py -

CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_TASK_DEFAULT_QUEUE = 'tk-test-queue'
CELERY_BROKER_URL = "sqs://%s:%s@" % (quote(os.environ.get('AWS_ACCESS_KEY_ID'), safe=''), quote(os.environ.get('AWS_SECRET_ACCESS_KEY'), safe=''))
CELERY_BROKER_TRANSPORT_OPTIONS = {
    'region': 'ap-south-1',
    'visibility-timeout': 60 * 30,
    'polling_interval': 1
}
CELERY_RESULT_BACKEND = None

My celery.py looks like this -

import os
from celery import Celery

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'my_project.settings')

app = Celery('my_project')
app.config_from_object('django.conf:settings', namespace="CELERY")
app.autodiscover_tasks()

I have defined a task in my_app/tasks.py -

from time import sleep, ctime

from celery import shared_task


@shared_task
def export_report_to_s3():
    print(f'Starting task at {ctime()}')
    sleep(25)
    print(f'Task finished at {ctime()}')

Which I am using in my my_app/views.py -

class TestCeleryView(View):
    def get(self, request):
        print('In view')
        export_report_to_s3.delay()
        return HttpResponse('Success')

I am starting the celery worker by running -

celery -A my_project worker -l INFO

Every time I visit the URL for my view function, I get a Success response from the view, and the number of in-flight requests on SQS goes up. But I get no output by the Celery worker. This is my first time working with Celery as well as SQS. What am I doing wrong?

Solution

I was seeing exactly the same issue with Celery 5.3.1.

I had to update Celery to >5.3.6 to get this fix:

Update kombu>=5.3.4 to fix SQS request compatibility with boto JSON serializer (#8646)

After this SQS messages actually got read and processed by Celery

https://docs.celeryq.dev/en/stable/changelog.html#version-5-3-6