rabbitmqcelerymessage-queuesystem-designrabbitmq-management

Celery creates 3 queues in RabbitMQ message queue


I was using celery as task queue and RabbitMQ as message queue, When pushing my tasks using the delay function to the queue. I see that there were 3 queues created in the rabbit mq. I don't understand what and why do we need these 2 extra queue. Also how do I identify onto which queue my tasks are actually getting pushed into?

Started celery :

celery -A myproject worker -l info

[tasks]
  . app1.tasks.add

[2022-06-10 06:16:14,132: INFO/MainProcess] Connected to amqp://himanshu:**@IPADDRESS/vhostcheck
[2022-06-10 06:16:14,142: INFO/MainProcess] mingle: searching for neighbors
[2022-06-10 06:16:15,165: INFO/MainProcess] mingle: all alone
[2022-06-10 06:16:15,182: WARNING/MainProcess] /etc/myprojectenv/lib/python3.8/site-packages/celery/fixups/django.py:203: UserWarning: Using settings.DEBUG leads to a memory
            leak, never use this setting in production environments!
  warnings.warn('''Using settings.DEBUG leads to a memory

[2022-06-10 06:16:15,182: INFO/MainProcess] celery@ubuntu-s-1vcpu-1gb-blr1-01 ready.
[2022-06-10 06:17:38,485: INFO/MainProcess] Task app1.tasks.add[be566921-b320-466c-b406-7a6ed7ab06e7] received
[2022-06-10 06:16:15,182: INFO/MainProcess] celery@ubuntu-s-1vcpu-1gb-blr1-01 ready.
[2022-06-10 06:17:38,485: INFO/MainProcess] Task app1.tasks.add[be566921-b320-466c-b406-7a6ed7ab06e7] received
[2022-06-10 06:19:18,544: INFO/ForkPoolWorker-1] Task app1.tasks.add[be566921-b320-466c-b406-7a6ed7ab06e7] succeeded in 100.05838803993538s: 13

SO whenever I run my celery worker I see these 3 queues being generated.

RabbitMQ Management

enter image description here

What are those 3 queue and what for is celery using them for?

Also since queues are basically persistent database and therefore persistent queues, so why do they get deleted when I stop my workers. I see there is only 1 queue here after I stop celery.

enter image description here


Solution

  • Celery documentation does not give any details about these queues, so people had to look for answeres in the Celery/Kombu source code. Here is one example: https://github.com/celery/celery/issues/6371#issuecomment-716839203