python-3.xdjango-qcroniter

Django-Q scheduling tasks every X seconds

I am using Django-Q to schedule a periodic simple task that has to be repeated every < 1 minute.

Croniter, used under the hood to parse cron expressions for the scheduler, specifies that cron "seconds" support is available:

Croniter seconds support

https://pypi.org/project/croniter/#about-second-repeats

So I created a cron-type schedule that looks like this:

    Schedule.objects.update_or_create(name='mondrian_scheduler', defaults= {'func':'mondrianapi.tasks.run_scheduler', 'schedule_type':Schedule.CRON,
     'cron': '* * * * * */20'} )

Django-q correctly parses and schedules the job, but the real frequency doesn't seem to go below 30 seconds (31, actually), whatever the 6th argument says:

2021-05-12 10:17:08.528307+00:00---run_bot ID 1
2021-05-12 10:17:39.166822+00:00---run_bot ID 1
2021-05-12 10:18:09.899772+00:00---run_bot ID 1
2021-05-12 10:18:40.648140+00:00---run_bot ID 1
2021-05-12 10:19:11.176563+00:00---run_bot ID 1
2021-05-12 10:19:41.857376+00:00---run_bot ID 1
Solution

  • The guard (or sentinel) process is responsible for querying for any scheduled tasks which are due, and it only does this twice per minute:

    Scheduler

    Twice a minute the scheduler checks for any scheduled tasks that should be starting.

    • Creates a task from the schedule
    • Subtracts 1 from django_q.Schedule.repeats
    • Sets the next run time if there are repeats left or if it has a negative value.

    https://django-q.readthedocs.io/en/latest/architecture.html?highlight=scheduler#scheduler

    The guard process is also responsible for checking that all of the other processes are still running, so it is not exactly thirty seconds.

    Unfortunately the scheduler interval is not configurable. If you're comfortable modifying django_q, the relevant code is in django_q/cluster.py, in Sentinel.guard().