prometheusprometheus-alertmanager

How to set alertmanager in prometheus-community/prometheus helm chart?


I'm using this chart:

https://artifacthub.io/packages/helm/prometheus-community/prometheus

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install my-prometheus prometheus-community/prometheus --values values.yaml --version 27.11.0

In the values.yaml, I set these serverFiles:

serverFiles:
  prometheus.yml:
    scrape_configs:
      - job_name: 'prometheus'
        static_configs:
          - targets:
              - localhost:9090
      - job_name: 'opencost'
        static_configs:
          - targets: ['opencost.opencost-exporter:9003']
  alerting_rules.yml:
    groups:
      - name: opencost-alerts
        rules:
          - alert: HighCost
            expr: node_total_hourly_cost > 1
            for: 1m
            labels:
              severity: critical
            annotations:
              description: "The cost is higher than expected for the last 1 minutes"
              summary: "Cost is over $1 for the last 1 minutes"
  alertmanager.yml:
    global:
      resolve_timeout: 5m
    route:
      receiver: 'slack-notifications'
      group_by: ['alertname']
      group_wait: 10s
      group_interval: 5m
      repeat_interval: 1h
    receivers:
      - name: 'slack-notifications'
        slack_configs:
          - api_url: 'https://hooks.slack.com/services/XXXXXXXX/YYYYYYYYY/ZZZZZZZZZZZZZZZZZZ'
            channel: '#alert-channel'
            send_resolved: true
            username: 'Prometheus'
            text: "{{ .CommonLabels.alertname }}: {{ .CommonAnnotations.description }}"

But when alert firing in Prometheus UI, there isn't any send log in alertmanager pod. And sure can't receive alert from Slack.

I'm sure the slack webhook is right. Where is wrong?

I want to receive alert in Slack.


Solution

  • This way works:

    alertmanager:
      config:
        global:
          resolve_timeout: 5m
        route:
          receiver: 'slack-notifications'
          group_by: ['alertname']
          group_wait: 10s
          group_interval: 5m
          repeat_interval: 1h
        receivers:
          - name: 'slack-notifications'
            slack_configs:
              - api_url: 'https://hooks.slack.com/services/XXXXXXXX/YYYYYYYYY/ZZZZZZZZZZZZZZZZZZ'
                channel: '#alert-channel'
                send_resolved: true
                username: 'Prometheus'
                text: "{{ .CommonLabels.alertname }}: {{ .CommonAnnotations.description }}"
    ...