Use Servicemonitor to check services via external hostname

We have a setup with external-DNS to create and bind dns entries based on service annotations.

For example we have a service for the alertmanager like this:

apiVersion: v1
kind: Service
metadata:
  name: prometheus-kube-prometheus-alertmanager
  namespace: prometheus
  labels:
...
    heritage: Helm
    prometheus-monitor-https: 'true'
    release: prometheus
    self-monitor: 'true'
  annotations:
    external-dns.alpha.kubernetes.io/hostname: alertmanager.ourdomain.com
    external-dns.alpha.kubernetes.io/ttl: '60'
spec:
  ports:
    - name: web
      protocol: TCP
      port: 80
      targetPort: 9093
      nodePort: 31126
  selector:
    alertmanager: prometheus-kube-prometheus-alertmanager
    app.kubernetes.io/name: alertmanager
  type: LoadBalancer
  sessionAffinity: None
  externalTrafficPolicy: Cluster

(abbreviated)

I want to use the blackbox exporter with the data from the annotations, so we don't have to manually add the monitoring here, but rather rely on kubernetes to provide the information what to monitor.

For that I wrote a servicemonitor, but it doesn't match the services and calls the blackbox exporter.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: blackbox-exporter-monitor-https-external
  namespace: prometheus
spec:
  namespaceSelector:
    any: true
  selector:
    matchLabels:
      prometheus-monitor-https: any
  targetLabels:
    - environment
    - instance
  endpoints:
    - metricRelabelings:
        - sourceLabels: [__meta_kubernetes_service_annotation_external_dns_alpha_kubernetes_io_hostname]
          targetLabel: __param_target
          replacement: "https://$1"
        - sourceLabels: [__param_target]
          targetLabel: instance
        - targetLabel: __param_scheme
          replacement: https
        - targetLabel: __address__
          replacement: prometheus-blackbox-exporter:9115
      path: /probe
      params:
        debug:
          - "true"
        module:
          - "http_2xx"

I am not seeing why it shouldn't match the service. Do you have any hints?

The service has label prometheus-monitor-https: 'true', while the ServiceMonitor has a selector.matchLabels of prometheus-monitor-https: any.

If you change this such that the selector.matchLabels of the ServiceMonitor equals prometheus-monitor-https: 'true', then I think it should work. The matchLabels looks for expected matches of the label key, value pair.

Also I see that you wrote namespaceSelector is any: true. It is good to know that the namespaceSelector works in a different way. It expects the labels of the namespace it should find the resource in. In your case it will look for a namespace that has the label any: true. But I think you actually want to select all namespaces, which is equal to not specifying a namespaceSelector at all.