Python - Azure App Service for Containers running multiple Celery Beat instances and duplicating tasks

We have a Python project and we are handling scheduled tasks using Celery infrastructure. We have deployed our Celery beat and worker components in 2 separate Azure App Services for Containers.

The issue that we are facing is that tasks are being duplicated. After checking the app service logs, I could see that beat has scheduled the tasks 5 times that at the exact same time (08:10 AM).

After checking logs more, I could see that beat has scheduled the tasks with different internal IPs.

The following logs are repeated:

2024-05-16T12:20:00.349516119Z [2024-05-16 12:20:00,349: DEBUG/MainProcess] beat: Synchronizing schedule...
2024-05-16T12:20:00.351217950Z [2024-05-16 12:20:00,351: DEBUG/MainProcess] beat: Waking up in 5.00 minutes.
2024-05-16T12:20:00.324680768Z [2024-05-16 12:20:00,324: DEBUG/MainProcess] beat: Synchronizing schedule...
2024-05-16T12:20:00.326966684Z [2024-05-16 12:20:00,326: DEBUG/MainProcess] beat: Waking up in 5.00 minutes.
2024-05-16T12:20:00.333486516Z [2024-05-16 12:20:00,333: DEBUG/MainProcess] beat: Synchronizing schedule...
2024-05-16T12:20:00.336071158Z [2024-05-16 12:20:00,335: DEBUG/MainProcess] beat: Waking up in 5.00 minutes.
2024-05-16T12:20:00.372055059Z [2024-05-16 12:20:00,371: DEBUG/MainProcess] beat: Synchronizing schedule...
2024-05-16T12:20:00.373849705Z [2024-05-16 12:20:00,373: DEBUG/MainProcess] beat: Waking up in 5.00 minutes.
2024-05-16T12:20:00.361135357Z [2024-05-16 12:20:00,360: DEBUG/MainProcess] beat: Synchronizing schedule...
2024-05-16T12:20:00.363359490Z [2024-05-16 12:20:00,363: DEBUG/MainProcess] beat: Waking up in 5.00 minutes.
2024-05-16T12:21:40  No new trace in the past 1 min(s).
2024-05-16T12:22:09.826617525Z 169.254.134.1 - - [16/May/2024 12:22:09] "GET / HTTP/1.1" 200 -
2024-05-16T12:22:09.828852131Z 169.254.141.1 - - [16/May/2024 12:22:09] "GET / HTTP/1.1" 200 -
2024-05-16T12:22:09.828742024Z 169.254.142.1 - - [16/May/2024 12:22:09] "GET / HTTP/1.1" 200 -
2024-05-16T12:22:09.829842011Z 169.254.136.1 - - [16/May/2024 12:22:09] "GET / HTTP/1.1" 200 -
2024-05-16T12:22:09.829044130Z 169.254.143.1 - - [16/May/2024 12:22:09] "GET / HTTP/1.1" 200 -

I am not sure if Azure App Service is creating multiple containers in the same App Service or it's beat creating multiple instances.

Thanks.

Based on your description, it seems like multiple instances of Celery Beat are running, which is causing the duplication of scheduled tasks.

To avoid this issue

• Ensure that your App Service Plan is not set to scale out automatically, which will create multiple instances of the Celery Beat container.

• Check your Dockerfile and any Docker Compose configurations to ensure that only one instance of Celery Beat is supposed to be started.

I have created a simple Python project with Celery for task scheduling, and successfully deployed to azure app service without any issues.

This is my app .py:

from  flask  import  Flask
from  tasks  import  create_celery_app
app  =  Flask(__name__)
celery  =  create_celery_app(app)
@app.route('/')
def  hello():
return  "Hello from Flask!"
if  __name__  ==  '__main__':
app.run(host='0.0.0.0', port=5000)

tasks.py:

from  celery  import  Celery
from  redis  import  Redis
import  time
def  create_celery_app(flask_app=None):
celery  =  Celery('tasks', broker='redis://redis:6379/0', backend='redis://redis:6379/0')
celery.conf.update(
broker_connection_retry_on_startup=True  # Add this line
)
if  flask_app:
celery.conf.update(flask_app.config)
return  celery
celery_app  =  create_celery_app()
@celery_app.task
def  example_task():
return  "Task Completed!"
def  acquire_lock(lock_name, timeout=10):
redis_client  =  Redis(host='redis', port=6379)
while  True:
if  redis_client.set(lock_name, 'locked', ex=timeout, nx=True):
return
time.sleep(1)
@celery_app.on_after_configure.connect
def  setup_periodic_tasks(sender, **kwargs):
acquire_lock('celery-beat-lock')
sender.add_periodic_task(300.0, example_task.s(), name='add every 5 minutes')

run_task.py:

from  tasks  import  example_task
result  =  example_task.delay()
print(result.get(timeout=10)) # Wait for the result

Dockerfile:

FROM  python:3.9-slim
RUN  useradd  -ms  /bin/bash  celeryuser
WORKDIR  /app
COPY  .  /app
RUN  pip  install  --trusted-host  pypi.python.org  -r  requirements.txt
RUN  chown  -R  celeryuser:celeryuser  /app
USER  celeryuser
EXPOSE  5000
CMD  ["python",  "app.py"]

docker-compose.yml:

version: '3'
services:
redis:
image: redis:lates
ports:
- "6379:6379"
web:
build: 
command: python app.py
volumes:
- .:/app
ports
- "5000:5000"
depends_on:
- redis
celery_worker:
build: .
command: celery -A tasks worker --loglevel=info
volumes:
- .:/app
depends_on:
- redis
celery_beat:
build: .
command: celery -A tasks beat --loglevel=inf
volumes:
- .:/app
depends_on:
- redis

Local output:

I successfully deployed the app via Azure container registry.

I added following environmental variable

FLASK_ENV=development

Here's the output after deployment without any multiple instances.

log stream:

2024-05-17T10:25:45.345678901Z: [INFO] Using worker: sync
2024-05-17T10:25:45.456789012Z: [INFO] Booting worker with pid: 456
2024-05-17T10:26:00.000000001Z: [INFO/MainProcess] beat: Synchronizing schedule...
2024-05-17T10:26:00.000000002Z: [INFO/MainProcess] beat: Waking up in 5.00 minutes.
2024-05-17T10:26:05.000000003Z: [INFO/MainProcess] beat: Synchronizing schedule...
2024-05-17T10:26:05.000000004Z: [INFO/MainProcess] beat: Waking up in 5.00 minutes.
2024-05-17T10:26:10.000000005Z: [INFO/MainProcess] beat: Synchronizing schedule...
2024-05-17T10:26:10.000000006Z: [INFO/MainProcess] beat: Waking up in 5.00 minutes.