I have a project using Celery with Redis backend to manage tasks. For local development I am trying to set up Celery with filesystem as broker instead of Redis. However, when I run Celery worker, it creates me control
folder in the root directory with the following contents:
/control
├── celery.pidbox.exchange
├── Q1.exchange
├── Q2.exchange
├── ...
I have trouble finding any resources on what it is and for what it is used exactly. My goal is to possibly move this folder to another location (ex. .celery/
folder), so that it does not sit in root directory.
Here is my Celery configuration:
class CeleryLocalConfig:
include = ["app.tasks"]
broker_url = "filesystem://"
result_backend = "file://.celery/broker/results"
broker_transport_options = {
"data_folder_in": ".celery/broker/out",
"data_folder_out": ".celery/broker/out",
"queue_order_strategy": "sorted"
}
celery_app = Celery(__name__)
celery_app.config_from_object(CeleryLocalConfig)
So far I have tried:
Running Celery worker from different directory than root, but then worker doesn't receive any tasks.
Providing --workdir
parameter when running Celery worker, but it seem to create different paths between Celery worker and Celery app and therefore fails.
Setting a control_folder
field did the trick for me:
CELERY_BROKER_TRANSPORT_OPTIONS = {
"data_folder_in": os.path.join(BASE_DIR, ".celery"),
"data_folder_out": os.path.join(BASE_DIR, ".celery"),
"control_folder": os.path.join(BASE_DIR, ".celery"),
}
So for you, I suppose, it would be:
broker_transport_options = {
"data_folder_in": ".celery/broker/out",
"data_folder_out": ".celery/broker/out",
"queue_order_strategy": "sorted",
"control_folder": ".celery/broker/out",
}