[SOLVED] How to delete celery task logs in flower?

How to delete celery task logs in flower?

I want to be able to delete tasks in the flower successful/failed tasks...

so my idea is to have a scheduled celery beat to delete tasks older than X amount of hours.

Anyone knows how to achieve this, where are the tasks stored... etc ?

Goal: Set a configuration variable that says number of hours let's say 48 hours of keeping the logs, then autodelete

That serves people mostly in europe with GDPR compliance, and also protects customers' privacy

Solution

I implemented new APIs for deleting tasks

Events:
  None delete_tasks_by_time(int to_timestamp)

Example: delete tasks older than 48 hours

import celery
import os
import timedelta
import datetime

broker = os.environ.get('CELERY_BROKER_URL')
app = celery.Celery('tasks', broker=broker)
flower = Flower(capp=app, options=flower_options)

time_delta = timedelta(hours=48)
now = datetime.datetime.now()
delete_before_time = now-time_delta

flower.events.delete_tasks_by_time(delete_before_time.timestamp())

Then we can a celery beat scheduler, that runs each hour, and delete the tasks

@app.task(queue='cleanup_tasks')
def clean_up_tasks():
    from flower.app import Flower

    time_delta = timedelta(hours=48)
    now = datetime.datetime.now()
    delete_before_time = now-time_delta
    flower_options = object()
    flower_options.db = 'flower'
    flower_options.persistent = True
    flower_options.purge_offline_workers = 1
    # note: use env vars better
    flower = Flower(capp=app, options=flower_options)
    flower.events.delete_tasks_by_time(delete_before_time.timestamp())

@app.on_after_configure.connect
def add_periodic(**kwargs):
    app.add_periodic_task(crontab(hour="*", minute=0), clean_up_tasks.s(), name='cleanup-tasks')

This is especially helpful for people who wants to maintain a good GDPR compliance, as you keep only data for a short time for basically debugging, everything is configurable to taste

References: https://github.com/mher/flower/issues/1189 https://github.com/mher/flower/pull/1188

Hopefully the PR gets merged soon!

Enjoy <3