[SOLVED] Prometheus using html content in alerts annotations and using it in email template

Prometheus using html content in alerts annotations and using it in email template

Our alert looks like:

ALERT alert_name
   condition
   FOR 30s
   LABELS {some labels}
   ANNOTATIONS {
       header = "<b> data is {{ $labels.label_name }} </b>"
   }

email template looks like:

{{ define "our_default_template" }}
{{range .Alerts}}
{{ .Annotations.header }}
{{ end }}
{{ end }}

alertmanager.yml looks like:

receivers:
- name: 'email-sender'
  email_configs:
  - to: "email address"
    send_resolved: true
    html: '{{ template "our_default_template" . }}'

templates:
- '<path to templates>/*tmpl'

We are getting emails but content is not right.

What we are getting in mail:

<b> data is label_value </b>

What we want:

data is label_value

So what we want is html output.

Can someone help with this?

Solution

Firstly I'd recommend against doing HTML on the Prometheus end, as that may cause maintenance issues as your system evolves.

It looks like Go's auto-escaping of HTML is what's happening here, so you need a way to tell Go's templating language that that's safe. The alertmanager doesn't have that function (Prometheus does), so I've filed https://github.com/prometheus/alertmanager/issues/314 for this.

Once that's fixed you'll be able to do {{ .Annotations.header | safeHtml }}