pythonazurecontainersazure-appservicecelerybeat

Python - Azure App Service for Containers running multiple Celery Beat instances and duplicating tasks

We have a Python project and we are handling scheduled tasks using Celery infrastructure. We have deployed our Celery beat and worker components in 2 separate Azure App Services for Containers.

The issue that we are facing is that tasks are being duplicated. After checking the app service logs, I could see that beat has scheduled the tasks 5 times that at the exact same time (08:10 AM).

After checking logs more, I could see that beat has scheduled the tasks with different internal IPs.

The following logs are repeated:

2024-05-16T12:20:00.349516119Z [2024-05-16 12:20:00,349: DEBUG/MainProcess] beat: Synchronizing schedule...
2024-05-16T12:20:00.351217950Z [2024-05-16 12:20:00,351: DEBUG/MainProcess] beat: Waking up in 5.00 minutes.
2024-05-16T12:20:00.324680768Z [2024-05-16 12:20:00,324: DEBUG/MainProcess] beat: Synchronizing schedule...
2024-05-16T12:20:00.326966684Z [2024-05-16 12:20:00,326: DEBUG/MainProcess] beat: Waking up in 5.00 minutes.
2024-05-16T12:20:00.333486516Z [2024-05-16 12:20:00,333: DEBUG/MainProcess] beat: Synchronizing schedule...
2024-05-16T12:20:00.336071158Z [2024-05-16 12:20:00,335: DEBUG/MainProcess] beat: Waking up in 5.00 minutes.
2024-05-16T12:20:00.372055059Z [2024-05-16 12:20:00,371: DEBUG/MainProcess] beat: Synchronizing schedule...
2024-05-16T12:20:00.373849705Z [2024-05-16 12:20:00,373: DEBUG/MainProcess] beat: Waking up in 5.00 minutes.
2024-05-16T12:20:00.361135357Z [2024-05-16 12:20:00,360: DEBUG/MainProcess] beat: Synchronizing schedule...
2024-05-16T12:20:00.363359490Z [2024-05-16 12:20:00,363: DEBUG/MainProcess] beat: Waking up in 5.00 minutes.
2024-05-16T12:21:40  No new trace in the past 1 min(s).
2024-05-16T12:22:09.826617525Z 169.254.134.1 - - [16/May/2024 12:22:09] "GET / HTTP/1.1" 200 -
2024-05-16T12:22:09.828852131Z 169.254.141.1 - - [16/May/2024 12:22:09] "GET / HTTP/1.1" 200 -
2024-05-16T12:22:09.828742024Z 169.254.142.1 - - [16/May/2024 12:22:09] "GET / HTTP/1.1" 200 -
2024-05-16T12:22:09.829842011Z 169.254.136.1 - - [16/May/2024 12:22:09] "GET / HTTP/1.1" 200 -
2024-05-16T12:22:09.829044130Z 169.254.143.1 - - [16/May/2024 12:22:09] "GET / HTTP/1.1" 200 -

I am not sure if Azure App Service is creating multiple containers in the same App Service or it's beat creating multiple instances.

Thanks.

Solution

  • Based on your description, it seems like multiple instances of Celery Beat are running, which is causing the duplication of scheduled tasks.

    To avoid this issue

    I have created a simple Python project with Celery for task scheduling, and successfully deployed to azure app service without any issues.

    This is my app .py:

    from  flask  import  Flask
from  tasks  import  create_celery_app
app  =  Flask(__name__)
celery  =  create_celery_app(app)
@app.route('/')
def  hello():
return  "Hello from Flask!"
if  __name__  ==  '__main__':
app.run(host='0.0.0.0', port=5000)

    tasks.py:

    from  celery  import  Celery
from  redis  import  Redis
import  time
def  create_celery_app(flask_app=None):
celery  =  Celery('tasks', broker='redis://redis:6379/0', backend='redis://redis:6379/0')
celery.conf.update(
broker_connection_retry_on_startup=True  # Add this line
)
if  flask_app:
celery.conf.update(flask_app.config)
return  celery
celery_app  =  create_celery_app()
@celery_app.task
def  example_task():
return  "Task Completed!"
def  acquire_lock(lock_name, timeout=10):
redis_client  =  Redis(host='redis', port=6379)
while  True:
if  redis_client.set(lock_name, 'locked', ex=timeout, nx=True):
return
time.sleep(1)
@celery_app.on_after_configure.connect
def  setup_periodic_tasks(sender, **kwargs):
acquire_lock('celery-beat-lock')
sender.add_periodic_task(300.0, example_task.s(), name='add every 5 minutes')

    run_task.py:

    from  tasks  import  example_task
result  =  example_task.delay()
print(result.get(timeout=10)) # Wait for the result

    Dockerfile:

    FROM  python:3.9-slim
RUN  useradd  -ms  /bin/bash  celeryuser
WORKDIR  /app
COPY  .  /app
RUN  pip  install  --trusted-host  pypi.python.org  -r  requirements.txt
RUN  chown  -R  celeryuser:celeryuser  /app
USER  celeryuser
EXPOSE  5000
CMD  ["python",  "app.py"]

    docker-compose.yml:

    version: '3'
services:
redis:
image: redis:lates
ports:
- "6379:6379"
web:
build: 
command: python app.py
volumes:
- .:/app
ports
- "5000:5000"
depends_on:
- redis
celery_worker:
build: .
command: celery -A tasks worker --loglevel=info
volumes:
- .:/app
depends_on:
- redis
celery_beat:
build: .
command: celery -A tasks beat --loglevel=inf
volumes:
- .:/app
depends_on:
- redis

    Local output: enter image description here

    enter image description here

    enter image description here enter image description here

    I successfully deployed the app via Azure container registry.

    I added following environmental variable

    FLASK_ENV=development

    enter image description here

    Here's the output after deployment without any multiple instances.

    enter image description here

    log stream:

    2024-05-17T10:25:45.345678901Z: [INFO] Using worker: sync
2024-05-17T10:25:45.456789012Z: [INFO] Booting worker with pid: 456
2024-05-17T10:26:00.000000001Z: [INFO/MainProcess] beat: Synchronizing schedule...
2024-05-17T10:26:00.000000002Z: [INFO/MainProcess] beat: Waking up in 5.00 minutes.
2024-05-17T10:26:05.000000003Z: [INFO/MainProcess] beat: Synchronizing schedule...
2024-05-17T10:26:05.000000004Z: [INFO/MainProcess] beat: Waking up in 5.00 minutes.
2024-05-17T10:26:10.000000005Z: [INFO/MainProcess] beat: Synchronizing schedule...
2024-05-17T10:26:10.000000006Z: [INFO/MainProcess] beat: Waking up in 5.00 minutes.