prometheus-alertmanager

Prometheus alert multiple routes and receiver doesn't send one rules


I have this alert configuration and expect this behavior.

If destination: bloom and severity: info send to slack-alert-info - it's work

Error there. If destination: bloom and severity: warning|critical send to slack-alert-multi - this works with error. Sverity: warning sending as expected to both Slack's channels but critical sending only to default channel.

May someone help me understand my error, please?

Amtool gives no error.

amtool config routes test --config.file=/opt/prometheus/etc/alertmanager.yml --tree --verify.receivers=slack-alert-multi severity=warning destination=bloom
Matching routes:
.
└── default-route
    └── {destination=~"^(?:bloom)$",severity=~"^(?:warning|critical)$"}  receiver: slack-alert-multi

slack-alert-multi


amtool config routes test --config.file=/opt/prometheus/etc/alertmanager.yml --tree --verify.receivers=slack-alert-multi severity=critical destination=bloom
Matching routes:
.
└── default-route
    └── {destination=~"^(?:bloom)$",severity=~"^(?:warning|critical)$"}  receiver: slack-alert-multi

slack-alert-multi

Alert configuration

...
    labels:
      alerttype: infrastructure
      severity: warning
      destination: bloom
...
---
global:
  resolve_timeout: 30m

route:
  group_by: [ 'alertname', 'cluster', 'severity' ]
  group_wait: 30s
  group_interval: 30s
  repeat_interval: 300s
  receiver: 'slack'

  routes:
  - receiver: 'slack-alert-multi'
    match_re:
      destination: bloom
      severity: warning|critical
  - receiver: 'slack-alert-info'
    match_re:
      destination: bloom
      severity: info

receivers:
- name: 'slack-alert-multi'
  slack_configs:
  - api_url: 'https://hooks.slack.com/services/T0/B0/V2'
    channel: '#alert-upload'
    send_resolved: true
    icon_url: 'https://avatars3.githubusercontent.com/u/3380462'
    title: '{{ template "custom_title" . }}'
    text: '{{ template "custom_slack_message" . }}'
  - api_url: 'https://hooks.slack.com/services/T0/B0/J1'
    channel: '#alert-exports'
    send_resolved: true
    icon_url: 'https://avatars3.githubusercontent.com/u/3380462'
    title: '{{ template "custom_title" . }}'
    text: '{{ template "custom_slack_message" . }}'

# Default receiver
- name: 'slack'
  slack_configs:
  - api_url: 'https://hooks.slack.com/services/T0/B0/2x'
    channel: '#aws-notification'
    send_resolved: true
    icon_url: 'https://avatars3.githubusercontent.com/u/3380462'
    title: '{{ template "custom_title" . }}'
    text: '{{ template "custom_slack_message" . }}'

- name: 'slack-alert-info'
  slack_configs:
  - api_url: 'https://hooks.slack.com/services/T0/B0/EA'
    channel: '#alert-info'
    send_resolved: true
    icon_url: 'https://avatars3.githubusercontent.com/u/3380462'
    title: '{{ template "custom_title" . }}'
    text: '{{ template "custom_slack_message" . }}'

templates:
- '/opt/alertmanager_notifications.tmpl'

Solution

  • Try add

    continue: true
    

    into

      - receiver: 'slack-alert-info'
        match_re:
          destination: bloom
          severity: info
          continue: true