I just installed prometheus operator as indicated here: https://github.com/prometheus-operator/kube-prometheus:
kubectl apply --server-side -f manifests/setup
kubectl wait \
--for condition=Established \
--all CustomResourceDefinition \
--namespace=monitoring
kubectl apply -f manifests/
After that I just tried to setup my own service monitor for grafana as follows:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: in1-grafana-service-monitor
namespace: monitoring
spec:
selector:
matchLabels:
app.kubernetes.io/name: grafana
endpoints:
- port: http
interval: 10s
This monitor works just fine and I can see it in the Prometheus /targets and /service-discovery.
The fact is that when I want to create this same service monitor but outside the "monitoring" namespace it just not appears neither in /targets or in /service-discovery. My setup for this service monitor is as follows:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: out1-grafana-service-monitor
namespace: other-namespace
spec:
selector:
matchLabels:
app.kubernetes.io/name: grafana
namespaceSelector:
any: true
endpoints:
- port: http
interval: 10s
How can I make Prometheus operator to scrape service monitors (and services) outside the monitoring namespace?
I checked the output of kubectl get prom -Ao yaml and it just displays an empty list:
[...]
serviceMonitorNamespaceSelector: {}
serviceMonitorSelector: {}
[...]
Any help will be appreciated.
Thank you.
I expect that the service monitor outside the monitoring namespace works as I need it for other service (Not for Grafana).
After looking at the yaml files I realized that Prometheus doesn't have the permissions to read all namespaces. And after looking at the repository customization examples I found the solution: https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/customizations/monitoring-additional-namespaces.md
Hope this helps someone else in the future.