I'm using this chart:
https://artifacthub.io/packages/helm/prometheus-community/prometheus
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install my-prometheus prometheus-community/prometheus --values values.yaml --version 27.11.0
In the values.yaml
, I set these serverFiles:
serverFiles:
prometheus.yml:
scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets:
- localhost:9090
- job_name: 'opencost'
static_configs:
- targets: ['opencost.opencost-exporter:9003']
alerting_rules.yml:
groups:
- name: opencost-alerts
rules:
- alert: HighCost
expr: node_total_hourly_cost > 1
for: 1m
labels:
severity: critical
annotations:
description: "The cost is higher than expected for the last 1 minutes"
summary: "Cost is over $1 for the last 1 minutes"
alertmanager.yml:
global:
resolve_timeout: 5m
route:
receiver: 'slack-notifications'
group_by: ['alertname']
group_wait: 10s
group_interval: 5m
repeat_interval: 1h
receivers:
- name: 'slack-notifications'
slack_configs:
- api_url: 'https://hooks.slack.com/services/XXXXXXXX/YYYYYYYYY/ZZZZZZZZZZZZZZZZZZ'
channel: '#alert-channel'
send_resolved: true
username: 'Prometheus'
text: "{{ .CommonLabels.alertname }}: {{ .CommonAnnotations.description }}"
But when alert firing in Prometheus UI, there isn't any send log in alertmanager pod. And sure can't receive alert from Slack.
I'm sure the slack webhook is right. Where is wrong?
I want to receive alert in Slack.
This way works:
alertmanager:
config:
global:
resolve_timeout: 5m
route:
receiver: 'slack-notifications'
group_by: ['alertname']
group_wait: 10s
group_interval: 5m
repeat_interval: 1h
receivers:
- name: 'slack-notifications'
slack_configs:
- api_url: 'https://hooks.slack.com/services/XXXXXXXX/YYYYYYYYY/ZZZZZZZZZZZZZZZZZZ'
channel: '#alert-channel'
send_resolved: true
username: 'Prometheus'
text: "{{ .CommonLabels.alertname }}: {{ .CommonAnnotations.description }}"
...