prometheuspromqlprometheus-node-exporter

PromQL and node-exporter: Peak memory consumption on a server during last minute

I try to obtain peak memory consumption on a server during last minute using PromQL and node-exporter metrics. When I use prometheus API like this:

curl -X POST -g 'http://epgt012:9070/api/v1/query?query=max_over_time(node_memory_MemTotal_bytes{instance="epgp003:9401"}-(node_memory_MemFree_bytes{instance="epgp003:9401"}+node_memory_Cached_bytes{instance="epgp003:9401"}+node_memory_Buffers_bytes{instance="epgp003:9401"})[1m])'

I get this error: {"status":"error","errorType":"bad_data","error":"invalid parameter \"query\": 1:143: parse error: unexpected identifier \"node_memory_Cached_bytes\""}

I should note that when I run curl -X POST -g 'http://epgt012:9070/api/v1/query?query=node_memory_Cached_bytes{instance="epgp003:9401"}' I get proper response.

Once I run that query inside prometheus UI:

max_over_time(node_memory_MemTotal_bytes{instance="epgp003:9401"}-(node_memory_MemFree_bytes{instance="epgp003:9401"}+node_memory_Cached_bytes{instance="epgp003:9401"}+node_memory_Buffers_bytes{instance="epgp003:9401"})[1m])

I get this error: Error executing query: invalid parameter "query": 1:268: parse error: ranges only allowed for vector selectors.

So it looks like I have two problems here. At the end I need to obtain results using curl. I tried all kind of different variations of the above, but couldn't get a working query.

Additionally it also would be nice to get peak memory consumption on a server between point in time - start and end (and not during last minute) using query_range api call.

Solution

  • Your original problem with curl

    + has a special meaning in URLs, and need to be encoded when used there.

    Your parameter query is expected to be

    max_over_time(node_memory_MemTotal_bytes{instance%3D"epgp003%3A9401"}-(node_memory_MemFree_bytes{instance%3D"epgp003%3A9401"}%2Bnode_memory_Cached_bytes{instance%3D"epgp003%3A9401"}%2Bnode_memory_Buffers_bytes{instance%3D"epgp003%3A9401"})[1m])

    You can achieve this by manually encoding your parameters, or using something like --data-urlencode

    Your promQL problem

    PromQL doesn't allow to apply range selector straight to the complex selector, it is only applicable to the vector selector. But you can use subquery syntax. In this case you query will look like this:

    max_over_time((node_memory_MemTotal_bytes{instance="epgp003:9401"}-(node_memory_MemFree_bytes{instance="epgp003:9401"}+node_memory_Cached_bytes{instance="epgp003:9401"}+node_memory_Buffers_bytes{instance="epgp003:9401"}))[1m:])