A Prometheus rules file like this:
groups:
- name: ./example.rules
rules:
- alert: ExampleAlert
expr: vector(1)
The Kubernetes prometheus-operator requires an extra layer of yaml configuration data and looks like this:
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
creationTimestamp: null
labels:
prometheus: example
role: alert-rules
name: prometheus-example-rules
spec:
groups:
- name: ./example.rules
rules:
- alert: ExampleAlert
expr: vector(1)
With files using the former, raw Prometheus rule format, I can use Prometheus promtool
to run unit tests. (see link 3 below)
With files using the latter, extended PrometheusRule format, I can use kubectl apply -f prometheus_rule_file.yaml
to load the rules into my prometheus-operator installation.
I want to write and maintain rules in one of these file formats and be able to use the same rule file both with promtool's unit test system and to load the same rule into a prometheus-operator installation. I don't see an easy way to do this.
Is there a way I can have a rule using the raw Prometheus rule format and load it into a prometheus-operator PrometheusRule without maintaining a separate redundant yaml file and without writing a customized tool to convert it?
The way I got this to work with helm is to create a separate chart and put rules files along with the chart files. This means you need to deploy them independently and perhaps that's what you intend.
Steps
helm create prometheus-rules
rules
directory and put prometheus rule files in that directory.templates/
directorytemplates/prometheus-rules.yaml
apiVersion: v1
kind: List
items:
{{- $root := .Files }}
{{- range $path, $bytes := $root.Glob "rules/**.yaml" }}
- apiVersion: "monitoring.coreos.com/v1"
kind: PrometheusRule
metadata:
name: prometheus-rules-{{ $path | base }}
labels:
app: prometheus-operator
spec:
groups:
{{ $root.Get $path | indent 8 }}
{{- end}}
NOTE: one tricky thing is to watch out for prometheus.prometheusSpec.ruleSelectorNilUsesHelmValues
value when installing the prometheus-operator chart. It will only load rules which have the same labels as were installed during prometheus-operator. You can get around that by setting it to false when installing prometheus-operator. See ruleSelector logic for more details.