Daemonised Celery worker throws error on 1 task when starting manually the worker does not

I am trying to fix an inconvenient issue I'm dealing with in my application. I set up a Celery daemon that seems to behave differently on 1 task within a chain, compared to a smooth process when starting the worker manually.

This task tries to download pictures from a list of URLs and save them, but the daemon throws some 'TimeLimitExceeded' at some point. I can definitely run the worker manually (e.g. within a screen) but then I'm loosing the flexibility of the daemon and its logs ...

To solve that issue, I set up the task with (bind=True) and I implemented a try/except that supposedly retries the task if this specific error happens (Cf. Celery doc.).

@celery.task(bind=True)
def fetch_img(self, datasetId):

    list_imgs = retrieve_imgs(datasetId)    # list of pair url + new filepath
    total = len(list_imgs)
    for p in range(len(list_imgs)):
        url = list_imgs[p][0]
        filepath = list_imgs[p][1]
        filename = os.path.basename(filepath)

        try:
            fetch_img = fetchUrl(url, filepath)
            if fetch_img[0] is True:            # download picture
                # message
                mesPic_part1 = '\n' + "# Url '" + url + "' successfully fetched"
                mesPic_part2 = '\n' + "--> File saved as '.../" + datasetId + '/' + filename + "'"
                list_parts = [mesPic_part1, mesPic_part2]
                downloaded += 1

            else:
                # get error message if download failed
                list_parts = [fetch_img[1]]

            # Message(s)
            for m in list_parts:
                log_message_line(m)

        except TimeLimitExceeded as exc:
            raise self.retry(countdown=60, exc=exc)

    return datasetId

But it has not improved ... when the issue occurs with the daemon, the logs give :

[2019-07-23 18:44:24,691: ERROR/MainProcess] Task handler raised error: TimeLimitExceeded(300.0,)
Traceback (most recent call last):
  File "/opt/some/path/app/venv/lib/python3.6/site-packages/billiard/pool.py", line 658, in on_hard_timeout
    raise TimeLimitExceeded(job._timeout)
billiard.exceptions.TimeLimitExceeded: TimeLimitExceeded(300.0,)
[2019-07-23 18:44:24,694: ERROR/MainProcess] Hard time limit (300.0s) exceeded for application.core.celery.etl.task_etl_fetchImg.fetch_img[5cdce7d5-6ab2-425b-a1dd-5d847e3d403e]

Have you experienced something similar? I'd be glad if you happen to have some tips ... thank you in advance !

Solution

You receive the TimeLimitExceeded exception because your task took longer than 300 seconds to execute. 300 Seconds is the default time a task can run.

You can increase the time_limit (in sec) of a task by doing this.

@celery.task(time_limit=20)
def mytask():
    do_something()

You can also set time-limits for a task in configuration by using:

CELERY_ANNOTATIONS = {'module.mytask': {'time_limit': 20}}