I am running multiple docker stacks all providing the infrastructure for a product. Internally those docker stacks are monitored using Prometheus and alert to a teams channel. I want all docker stacks alerting to the same teams channel but need a way to tell from which server the notification came.
For example with my rule
- alert: InstanceDown
expr: up == 0
for: 0m
labels:
severity: critical
I am generating alerts for instances that go down. But since I have multiple stacks where all the services have the same internal host-names and IPs the alert generated and send to teams is missing the information from which server the alert came.
I was thinking to maybe merge some node-exporter information like node_dmi_info into the alert expr (up ==0), but I was not able to figure that out yet.
And I am also assuming that the alert only has access to the results of its expression so that I can't simply add something like
labels:
host: "{{ $labels.product_name }}"
trying to access node-exporter information.
I ended up injecting the hostname coming from the hostname command into prometheus and using that for the alerts by using the flag: --enable-feature=expand-external-labels and the following config:
global:
...
external_labels:
cluster: ${CLUSTER_NAME}
rule_files:
- rules.yml
alerting:
alert_relabel_configs:
- source_labels: [cluster]
action: replace
regex: (.*)
replacement: "$1"
target_label: cluster
alertmanagers:
- static_configs:
- targets:
- 'prometheus-alert:9093'
as explained here: https://www.lisenet.com/2021/use-external-labels-with-prometheus-alerts/