[SOLVED] Remove a part of a log in Loki

Remove a part of a log in Loki

I have installed Grafana, Loki, Promtail and Prometheus with the grafana/loki-stack.

I also have Nginx set up with the Nginx helm chart.

Promtail is ingesting logs fine into Loki, but I want to customise the way my logs look. Specifically I want to remove a part of the log because it creates errors when trying to parse it with either logfmt or json (Error: LogfmtParserErr and Error: JsonParserErr respectively).

The logs look like this:

2022-02-21T13:41:53.155640208Z stdout F timestamp=2022-02-21T13:41:53+00:00 http_request_method=POST http_response_status_code=200 http_response_time=0.001 http_version=HTTP/2.0 http_request_body_bytes=0 http_request_bytes=63

and I want to remove the part where it says stdout F so the log will look like this:

2022-02-21T13:41:53.155640208Z timestamp=2022-02-21T13:41:53+00:00 http_request_method=POST http_response_status_code=200 http_response_time=0.001 http_version=HTTP/2.0 http_request_body_bytes=0 http_request_bytes=63

I have figured out that on the ingestion side it could be something with Promtail, but ist it also possible to make a LogQL query in Loki to just replace that string? And how would one set up the Promtail configuration for the wanted behaviour?

Solution

Promtail should be configured to replace the string with the replace stage.

Here is a sample config that removes the stdout F part of the log for all logs coming from the namespace ingress.

promtail:
  enabled: true
  pipelineStages:
  - docker: {}
  - match:
      selector: '{namespace="ingress"}'
      stages:
      - replace:
          expression: "(stdout F)"
          replace: ""

Specifically this example works for the grafana/loki-stack chart.