[SOLVED] Prometheus push gateway only returns the last value instead of all

Prometheus push gateway only returns the last value instead of all

I use the push gateway from prometheus (https://prometheus.io/docs/practices/pushing/). And I push some metrics from a React UI very often like this:

# HELP prom_react_app_availability_total Application availability counter
# TYPE prom_react_app_availability_total counter
prom_react_app_availability_total{owner="test", status="started", timestamp="2022-09-13T06:48:19.280Z"} 1

and sometimes I have a "failed" status after "started".

# HELP prom_react_app_availability_total Application availability counter
# TYPE prom_react_app_availability_total counter
prom_react_app_availability_total{owner="test", status="failed", timestamp="2022-09-13T06:52:19.280Z"} 1

But when I take a look on the gateway's /metrics endpoint, then I only see the last value instead of all the submitted values over time. My Prometheus server collects data from this endpoint every 30s and so I lost a lot of data and maybe also the "failed", that occurred because the next push was a status="started". I thought, I see all events or a summary with increased values.

What is wrong?

Solution

Prometheus Pushgateway remembers only the last pushed sample per each time series by design. See these docs. If you want counting events when your app was available/unavailable, then take a look at statsd_exporter.

Side suggestion: it isn't recommended storing timestamps in metric labels, since every unique timestamp results in a new time series. This may slow down Prometheus and increase its memory usage significantly. Such a condition is named high churn rate and/or high cardinality. See also these docs.