Helm Prometheus operator - set email notifications/edit secrets

I have installed prometheus-operator via helm and now want to set email notifications for Alertmanager:

I edited Prometheus operator secret:

    kubectl get secret alertmanager-prometheus-prometheus-oper-alertmanager -n monitoring -o yaml
    apiVersion: v1
    data:
      alertmanager.yaml: Z2xvYmFsOgogIHJlc29sdmVfdGltZW91dDogNW0KcmVjZWl2ZXJzOgotIG5hbWU6ICJudWxsIgpyb3V0ZToKICBncm91cF9ieToKICAtIGpvYgogIGdyb3VwX2ludGVydmFsOiA1bQogIGdyb3VwX3dhaXQ6IDMwcwogIHJlY2VpdmVyOiAibnVsbCIKICByZXBlYXRfaW50ZXJ2YWw6IDEyaAogIHJvdXRlczoKICAtIG1hdGNoOgogICAgICBhbGVydG5hbWU6IFdhdGNoZG9nCiAgICByZWNlaXZlcjogIm51bGwi
    kind: Secret
    metadata:
      creationTimestamp: "2020-03-23T09:19:57Z"
      labels:
        app: prometheus-operator-alertmanager
        chart: prometheus-operator-8.12.2
        heritage: Helm
        release: prometheus
      name: alertmanager-prometheus-prometheus-oper-alertmanager
      namespace: monitoring
      resourceVersion: "1853097"
      selfLink: /api/v1/namespaces/monitoring/secrets/alertmanager-prometheus-prometheus-oper-alertmanager
      uid: eb7f514e-6bf4-4791-9c1a-e45590ba2a36
    type: Opaque



 echo 'Z2xvYmFsOgogIHJlc29sdmVfdGltZW91dDogNW0KcmVjZWl2ZXJzOgotIG5hbWU6ICJudWxsIgpyb3V0ZToKICBncm91cF9ieToKICAtIGpvYgogIGdyb3VwX2ludGVydmFsOiA1bQogIGdyb3VwX3dhaXQ6IDMwcwogIHJlY2VpdmVyOiAibnVsbCIKICByZXBlYXRfaW50ZXJ2YWw6IDEyaAogIHJvdXRlczoKICAtIG1hdGNoOgogICAgICBhbGVydG5hbWU6IFdhdGNoZG9nCiAgICByZWNlaXZlcjogIm51bGwi' | base64 --decode

Created a new alertmanager.yaml file:

global:
  resolve_timeout: 5m
route:
  group_by: [Alertname]
  # Send all notifications to me.
  receiver: email-alert

  group_by: ['job', 'alertname', 'service', 'severity']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 12h
  receiver: email-alert
  routes:
  - match:
      severity: critical
    receiver: email-alert

receivers:
- name: email-alert
  email_configs:
  - to: email@example.com
    from: email@example.com
    # Your smtp server address
    smarthost: smtp.office365.com:587
    auth_username: email@example.com
    auth_identity: email@example.com
    auth_password: Pass

Created template:

apiVersion: v1
data:
  alertmanager.yaml: ALERTMANAGER_CONFIG
kind: Secret
metadata:
  name: alertmanager-main
  namespace: monitoring
type: Opaque

Encoded it and applied:

sed "s/ALERTMANAGER_CONFIG/$(cat alertmanager.yaml | base64 -w0)/g" alertmanager-secret-k8s.yaml | kubectl apply -f -

I exposed Alertmanager as Nodeport via port 30700.

So when I access it (http://IP:30700/#/status), I see this alertmanager.yaml is not applied, i.e., secret is not changed.

What needs to be done so I can edit this Prometheus Alertmanager secret?

tried with

helm upgrade prometheus stable/prometheus-operator --namespace monitoring -f alertmanager.yaml 

without help

Figured it out thanks to this reference

• delete current secret:

  kubectl delete secret alertmanager-prometheus-prometheus-oper-alertmanager -n monitoring
  

• create file alertmanager.yaml:

    global:
      resolve_timeout: 5m
    route:
      receiver: 'email-alert'
      group_by: ['job']
  
  
      routes:
      - receiver: 'email-alert'
        match:
          alertname: etcdInsufficientMembers
        group_wait: 30s
        group_interval: 5m
        repeat_interval: 12h  
  
    receivers:
    - name: email-alert
      email_configs:
      - to: receiver@example.com
        from: sender@example.com
        # Your smtp server address
        smarthost: smtp.office365.com:587
        auth_username: sender@example.com
        auth_identity: sender@example.com
        auth_password: pass
  

• create new secret with same name as old one:

  kubectl create secret generic alertmanager-prometheus-prometheus-oper-alertmanager -n monitoring --from-file=alertmanager.yaml