openshiftprometheusgrafanapromqlthanos

Displaying multiple OpenShift clusters version within grafana


I want to make a pie chart where I display the current versions of available OpenShift clusters. The clusters uses remote_write to a central Thanos instance, from where we can use promql to query metrics.

The metric in question here should be cluster_version, and the query would be something like,

group by (version,cluster) (cluster_version{type="completed"}) which in a pie chart diagram would look like,

what i have

given that we have 7 clusters available.

Now that is not what I want, what I rather want is something like this, what i want

where each part of the pie represents the versions, and the tooltip would be a commaseparated list of clusters having that version (or even nicer, each cluster on a separate row). The main point here is to have only the amount of slices that there are versions available (not as in the first chart where we have pies for each combination of version+cluster).

I'm not sure if this is possible or not, but I figure given that we use the combination of the tools prometheus, thanos and grafana, there should be some way of getting what I want.

The metric I currently use is defined here, https://github.com/openshift/cluster-version-operator/blob/db8190f7c0386ce93e9730644a70c263ef276c5e/pkg/cvo/metrics.go#L68, but I can also create custom recording rules if that is needed.

I'm not keen on installing external Grafana Plugins, so I would like to accomplish the expected results given the "default functionality" from within the tools itself.

Please explain to me how I can accomplish this, and if it is not possible, why it's not possible. Even though I'm not five years old, try to explain it to me like I was, because I find this quite hard to grasp and I'm having trouble understanding concepts such as time series, tables format, fields, lables, matrix and what not.

According to a couple of hours with chatgpt, this is not possible - is that really true ?

A single dataframe response from the query looks like the following,

[
  {
    "schema": {
      "refId": "B",
      "meta": {
        "type": "timeseries-multi",
        "typeVersion": [
          0,
          1
      ],
        "custom": {
          "resultType": "matrix"
        }
      },
      "fields": [
        {
          "name": "Time",
          "type": "time",
          "typeInfo": {
            "frame": "time.Time"
          },
          "config": {
            "interval": 15000
          }
        },
        {
          "name": "cluster_version",
          "type": "number",
          "typeInfo": {
            "frame": "float64"
          },
          "labels": {
            "__name__": "cluster_version",
            "cluster": "cluster1",
            "endpoint": "metrics",
            "from_version": "4.16.38",
            "image": "quay.io/openshift-release-dev/ocp-release@sha256:2754cd66072e633063b6bf26446978102f27dd19d4668b20df2c7553ef9ee4cf",
            "job": "cluster-version-operator",
            "kluster": "cluster1",
            "namespace": "openshift-cluster-version",
            "pod": "cluster-version-operator-55fc6649f6-b4d4h",
            "prometheus": "openshift-monitoring/k8s",
            "service": "cluster-version-operator",
            "tenant_id": "default-tenant",
            "type": "completed",
            "version": "4.16.39"
          },
          "config": {
            "displayNameFromDS": "4.16.39 | cluster1"
          }
        }
      ]
    },
    "data": {
      "values": [
        [
          1747838670000,
          1747838685000,
          1747838700000,
          1747838715000,
          1747838730000,
          1747838745000,
          1747838760000,
          1747838775000,
          1747838790000
        ],
        [
          1746492425,
          1746492425,
          1746492425,
          1746492425,
          1746492425,
          1746492425,
          1746492425,
          1746492425,
          1746492425
        ]
      ]
    }
  }
]

Solution

  • It is possible.

    1. Create Pie chart panel with your query
    2. In Options menu (just under query) change Format to Table and Type to Instant
    3. Add transformation Group by
      • Time: ignored
      • version: Group by
      • cluster: Calculate > All values
      • Value: Calculate > Total
    4. In panel options (side panel): Value options > Show > All values

    Result will look something like that: Example of panel

    Explanation: biggest trick here is query options. Type makes so that you are only getting latest (relative to dashboard time) value of the series and not all values. And Format converts time series into normal table, onto which transformations can be easily applied. Transformation then just makes to grouping as you wanted.


    I don't know how to make colours unique for versions though. As I understand, in my setup color is selected for number of items with same version, and nothing of what I tried changes that.