I struggle to implement a clean solution for an "delete all" button for our application while using the loki api.
Setup: Loki in monolithic mode in a cluster.
Goal: Loki accepts one api request that triggers deletion of all its stored logs.
After the /loki/api/v1/delete api has been enabled via configuration as described in their docs, i have to choose a deletion_mode.
filter-only says:
log lines matching the query in the delete request are filtered out when querying Loki. They are not removed from storage.
I have trouble understanding this. Does
filtered out when querying Loki
really mean, they will not show up when you query them, while they are still stored? This would be weird, i suppose that interpretation is wrong. So i assumed it meant, the queried data will be kept while everything else gets deleted.
One quick test:
curl -g -X POST 'http://loki-gateway.xxx.svc.cluster.local:PORT/loki/api/v1/delete?query={label="YYY"}&start=1591616227' -H 'X-Scope-OrgID: 1'
did teach me otherwise. Nothing was deleted, but the request has been received according to:
GET /loki/api/v1/delete`
The query {label="YYY"} has been chosen in various way. Either to an imaginary label or to a real one.
At this point, i assume i lack understanding of this endpoint.
filter-and-delete says:
log lines matching the query in the delete request are filtered out when querying Loki, and they are also removed from storage.
Here i would need to query the streams i want to be deleted. I would not consider this a clean solution for me, since either adding a stream would require this request to be expanded by it's label or i would need some omnipresent label for each new stream for deletion query purpose only.
Either solution seemed to be scalable for me. But i tested it with:
curl -g -X POST 'http://loki-gateway.xxx.svc.cluster.local:PORT/loki/api/v1/delete?query={real_label="real_value"}&start=1591616227' -H 'X-Scope-OrgID: 1'
and it worked.
This approach would be fine if i could use something like {*~="+*"} to cover every stream. But this does not seem to be possible (on stream selector level), or is it?
Does
filtered out when querying Loki
really mean, they will not show up when you query them, while they are still stored?
Yes it does mean exactly that. This is not "weird", because deletion is rather costly operation, and in case you need to only delete a couple lines, physical removal of stored data might be unreasonable.
I believe for your case of "delete all" solution this is not the right approach.
Regarding stream selector: you in fact need to use some label for selection.
Usually there are plenty labels to assist with this task. The best candidate, I believe, is label job, with selector {job!=""}.
If your records completely lack any "omnipresent label", you have two choices: use a list of labels, so that at least one of labels in that list is present in every record, and create tasks for deletion by looping this list. Or setup local to Loki solution, that will be wiping out it's storage.