pythoncelerycelerybeat

How not to start same task and wait until it is finished with celery beat

I have scheduled task with celery beat to run every 3 hours:

'sync_stuff': {
    'task': 'celery_tasks.sync_stuff',
    'schedule': crontab(hour='*/3')
}

Sometimes it takes longer than 3 hours to finish the task and I want to ensure that celery does not schedule and run the task again while the old instance is still running.

Is there a way to do that just with celery or celerybeat settings?

Solution

  • Unfortunately, you have to implement a locking strategy yourself.

    Read this part of the doc for additional details:

    Like with cron, the tasks may overlap if the first task doesn’t complete before the next. If that’s a concern you should use a locking strategy to ensure only one instance can run at a time (see for example Ensuring a task is only executed one at a time).

    Sources:

    http://docs.celeryproject.org/en/latest/userguide/periodic-tasks.html#entries http://docs.celeryproject.org/en/latest/tutorials/task-cookbook.html#cookbook-task-serial