djangocroncelerydjango-celerydjango-cron

Difference between usage of Django celery and Django cron-jobs?


I am sorry if its basics but I did not find any answers on the Internet comparing these two technologies. How should I decide when to use which as both can be used to schedule and process periodic tasks.

This is what an article says:

Django-celery :

Jobs are essential part of any application that does some processing for you in the background. If your job is real time Django application celery can be used.

Django-cronjobs :

django-cronjobs can be used to schedule periodic_task which is a valid job. django-cronjobs is a simple Django app that runs registered cron jobs via a management command.

can anyone explain me the difference between when should I choose which one and Why? Also I need to know why celery is used when the computing is distributed and why not cron jobs


Solution

  • The two things can be used for the same goal (background execution). However, if you are going to choose wisely, you should really understand that they are actually completely different things.

    Here's what I wish someone had told me back when I was a noob (instead of the novice level that I have achieved today :)).

    cron

    The concept of a cron job is that we want a command / process to be executed on some schedule. Furthermore, we want that process to receive x,y,z parameters, run with a,b,c environment variables, and as user id 123.

    Some cron systems may facilitate a few extra features, such as:

    For the most part, cron systems are meant to be dumb: "just run this command at this time, thanks!".

    Celery

    The concept of Celery is much more sophisticated. It works with tasks, chains & chords of tasks, error handling, and (in most cases) collection of work result. It has a queue (or many queues) of work and a worker (or many). When a task (really just a message describing requested work) enters the queue it waits there until a worker is available to handle it. Much the same way as 1 or more employees at the DMV service a room full of waiting customers.

    Furthermore, Celery can facilitate distributed work. That's a bit like (if I may torture the analogy a bit) - the difference between a DMV office where every worker shares the same phone, computer, copier, etc and a DMV where workers have dedicated resources and are never blocked by other workers.

    Celery for web apps

    In web applications, Celery is often used when a bit of web access results in a thing to be done that should be handled out of band of the conversation with the web browser. For example:

    Using Celery as cron

    When you use Celery as a "cron system" all you are really doing is saying: "hey, can someone please generate work of X type on Y schedule". A process is created that runs continuously which sleeps most of the time and wakes up occasionally to inject a bit of work into the queue on the schedule you requested.

    Usually the "hey someone" that you ask to do that for you is: celery beat and beat gets the schedule you want from somewhere in the database or from your settings file.