I've been struggling to get correct format for handling timestamp in promtail config.
Format of my log:
2022-08-02 16:46:02.141 content
My promtail config:
pipeline_stages:
- multiline:
firstline: '^\d{4}-\d{2}-\d{2}'
max_wait_time: 10s
- regex:
expression: '(?P<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3})(.*)$'
- labels:
time:
- timestamp:
source: time
format: 2006-01-02 15:04:05,999
With this conf I'm still getting "ts" field in Grafana with incorrect value (the value doesn't match with the value from log). I don't know if the format is correct, I tried many combinations but nothing worked as expected. In go playground the value was parsed as "2022-08-02 16:46:02.141 +0000 UTC" without errors - it it ok? How to make the "ts" value in grafana same as the value in log?
I tried debug mode yesterday and in promtail log was error like:
level=error ts=2022-08-03T12:37:14.977444831Z caller=client.go:380 component=client host=localhost:3100 msg="final error sending batch" status=400 error="server returned HTTP status 400 Bad Request (400): entry for stream '{filename=\"/opt/loki/my.log\", job=\"varlogs\", stream=\"stdout\", time=\"2022-08-03 14:38:49,150\"}' has timestamp too new: 2022-08-03T14:38:49Z"
I thought that loki can handle this (unordered_writes as true by default), is this problem of my format or loki conf?
Any help would be greatly appreciated.
I figured it out, this https://github.com/grafana/loki/issues/2639 helped me a lot. It was all about location in timestamp stage.
- timestamp:
source: time
format: 2006-01-02 15:04:05,999
location: "Europe/Berlin"