pythoncachingmemcachedcelerypylibmc

Disable caching on celery task

I have a celery chain that has multiple tasks, one of which is extracting a tar file and returning the JSON found within that tar file. At completion of the chain, pylibmc raises error 37 from memcached_set: SUCCESS which means that the memcached value associated with that task key is greater than 1MB.

This error does not interfere with the data integrity of the chain, but I'd really prefer to not be writing this data to the cache since it will never actually be used again.

I looked through the Celery documentation, but found nothing that would help with disabling the cache on a specific task. Any assistance would be appreciated.

Solution

  • You can use CELERY_IGNORE_RESULT or Task.ignore_result to not store the result. The value is still returned to the functions in the chain, they just aren't persisted to the cache. Here is an example using ignore_result

    @task(ignore_result=True)
def your_task():
    # your code here

    Docs: http://docs.celeryproject.org/en/latest/reference/celery.app.task.html#celery.app.task.Task.ignore_result

    Blog post which pointed me in the right direction: https://www.caktusgroup.com/blog/2014/09/29/celery-production/