kubernetesmonitoringprometheusthanos

Missing data in Thanos at specific resolution

Question

Problem: When querying Thanos for a graph at resolution lower than a certain threshold, Thanos will return data with missing data points when Prometheus does not (see screenshots later).

Setup: A kubernetes cluster. Thanos’s querier is configured to query a single Prometheus’s sidecar (I did this as a test, to be certain there would be no downsampling at play here.) The querier is launched without the --query.auto-downsampling option.

Question: Why isn’t Thanos returning all the data Prometheus does?

Graphs

Thanos Query for a 12h range, with a 90s resolution

Thanos Query for a 12h range, with a 90s resolution

Same Query, but on Prometheus

Prometheus Query for a 12h range, with a 90s resolution

Thanos Query for a 6h range, with a 90s resolution

Thanos Query for a 6h range, with a 90s resolution

(We get the same result on Prometheus)

Solution

  • Fixed it by upgrading Thanos from 0.19.0-rc.0 to 0.20.0…