I have log lines that contain a few timestamp fields. Here is an example of log line I am filtering in order to process it:
{
"time": "2022-06-22T10:33:08.710037238Z",
"@version": "1",
"message": "Duration at response processing ends",
"logger_name": "com.mks.cloud.filters.MainFilter",
"thread_name": "reactor-http-epoll-1",
"level": "INFO",
"level_value": 20000,
"rqst_id_ts": "b65c37d9284584e71b1dcd84b6a74075",
"rqst_end_ts": "1655893988698",
"rqst_start_ts": "1655893988698",
"rsp_start_ts": "1655893988709",
"rsp_end_ts": "1655893988709"
}
What I would like to do is calculate a value that would represent the duration between 2 timestamps in the log line so that I can then put the obtained range into quantile_over_time() or any other aggregate function.
For instance, the following works:
quantile_over_time(0.99, {app="myapp"} |~ "rsp_end_ts"
| json
| __error__ = ""
| unwrap rsp_end_ts | __error__="" [5m]) by (tsNs)
However this is not what I want to do since calculating the quantile of epoch timestamps makes no sense. What I want to calculate is the p99 of (rsp_end_ts - rqst_start_ts).
I tried the following which of course doesn't work, but gives an idea as to what I am attempting to do:
quantile_over_time(0.99, {app="myapp"} |~ "rsp_end_ts"
| json
| __error__ = ""
| unwrap (rsp_end_ts - rqst_start_ts) | __error__="" [5m]) by (tsNs)
If somehow there was a way to create a new label like rqst_duration=(rsp_end_ts - rqst_start_ts)
Then the following would be what I am looking for:
quantile_over_time(0.99, {app="myapp"} |~ "rsp_end_ts"
| json
| __error__ = ""
| unwrap rqst_duration | __error__="" [5m]) by (tsNs)
I couldn't find any documentation about this, which is very surprising, I would think (but it seems I might be wrong) this to be a common use case. Any help would be greatly appreciated :).
You can use template functions for that. Here's a sample query on the Grafana playground that you can inspire from.
So your query will look something like:
{app="myapp"} |~ "rsp_end_ts"
| json
} label_format result=`{{sub .rsp_end_ts rqst_start_ts}}` | line_format "{{ .result }}"
Then you can use the result.