I am trying to calculate the number of events (in my example deployments) per day. What I'm doing currently is that I'm sending the following counter events based on the HTTP API of pushgateway's
# TYPE deployments_count counter
# HELP deployments_count Deployments Counter
deployments_count{label1="${label1}",label2="${label2}"} 1
What I would like to calculate on my dashboard is how many distinct events (ie deployments) have taken place on a given day.
Empirically, I know that I should be having more than 10 events (deployments) per hour, but when I execute the following query I keep receiving 0
rate(deployments_count[24h])
Note that my counter always report 1 when a given event (deployment) has happened.
First of all, the rate() function calculate the per-second rate of increase of a counter. That is, even if your counter values were accurate, you would get the number of deployments happening per second (during the past 24 hours), and not per day.
If you want to calculate the number of deployments during the past 24 hours, there's the increase() function: increase(deployments_count[24h]).
But the reason that your current expression yields 0, is that the counter value always is 1. A counter must be incremented on every occurrence of an event (see Prometheus docs).
That is, you must somehow keep track of the current value of the counter and increment it on every deployment before pushing it to the Pushgateway, rather than just pushing 1 on every event. The latter approach doesn't work and it looks to Prometheus as if the value never changes.
There are two possible approaches for addressing this:
1. Not using a Pushgateway
Are you sure you need a Pushgateway or can you incorporate a Prometheus client library in your code? Check When to use the Pushgateway, and, in particular, a Pushgateway is not a distributed counter. In essence, the use case for a Pushgateway is for ephemeral jobs that need to deposit their metrics somewhere before they terminate.
If your code is permanently running, on the other hand, a Prometheus client library takes care of the counter incrementation logic and exposes the metric so that it can be directly scraped by Prometheus.
2. Keep track of counter value
If you must use a Pushgateway, you need to keep track of the current counter value so that you can increment it. You can either do it in your code, or query the current value from the Pushgateway itself, increment it, and push it back. Both of these approaches run into problems when there are multiple processes contributing to the counter (i.e. concurrent updates, race conditions).