I'm studying Prometheus Operator from the project: https://github.com/prometheus-operator/kube-prometheus
When my service/deployment is in the same "monitoring" namespace as prometheus I have success to visualize metrics. Example:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: caixa-app
namespace: monitoring
labels:
app.kubernetes.io/name: kube-caixa
app.kubernetes.io/part-of: kube-prometheus
spec:
endpoints:
- interval: 15s
port: http
selector:
matchLabels:
app: caixa
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: caixa-deployment
namespace: monitoring
spec:
replicas: 2
selector:
matchLabels:
app: caixa-pod
template:
metadata:
annotations:
prometheus.io/scrape: "true"
prometheus.io/path: /metrics
prometheus.io/port: "80"
labels:
app: caixa-pod
spec:
containers:
- name: caixa
image: 'REGISTRY-NAME/caixa:1.0'
ports:
- containerPort: 80
resources:
requests:
memory: "64Mi"
cpu: "250m"
limits:
memory: "128Mi"
cpu: "500m"
imagePullSecrets:
- name: regcred
---
apiVersion: v1
kind: Service
metadata:
name: caixa-svc
namespace: monitoring
labels:
app: caixa
spec:
selector:
app: caixa-pod
ports:
- name: http
port: 80
targetPort: 80
- name: https
port: 443
targetPort: 80
type: ClusterIP
In the way mentioned above I have success in the same namespace "monitoring"
However I need my Prometheus-operator to collect metrics from other namespaces that I will create. So I used the parameter "namespaceSelector".
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: caixa-app
namespace: monitoring
labels:
app.kubernetes.io/name: kube-caixa
app.kubernetes.io/part-of: kube-prometheus
spec:
endpoints:
- interval: 15s
port: http
namespaceSelector:
matchNames:
- shop
selector:
matchLabels:
app: caixa
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: caixa-deployment
namespace: shop
spec:
replicas: 2
selector:
matchLabels:
app: caixa-pod
template:
metadata:
annotations:
prometheus.io/scrape: "true"
prometheus.io/path: /metrics
prometheus.io/port: "80"
labels:
app: caixa-pod
spec:
containers:
- name: caixa
image: 'REGISTRY-NAME/caixa:1.0'
ports:
- containerPort: 80
resources:
requests:
memory: "64Mi"
cpu: "250m"
limits:
memory: "128Mi"
cpu: "500m"
imagePullSecrets:
- name: regcred
---
apiVersion: v1
kind: Service
metadata:
name: caixa-svc
namespace: shop
labels:
app: caixa
spec:
selector:
app: caixa-pod
ports:
- name: http
port: 80
targetPort: 80
- name: https
port: 443
targetPort: 80
type: ClusterIP
That way the metrics don't appear in prometheus, but they are in my Prometheus config.
- job_name: serviceMonitor/monitoring/caixa-app/0
honor_timestamps: true
scrape_interval: 15s
scrape_timeout: 10s
metrics_path: /metrics
scheme: http
follow_redirects: true
enable_http2: true
relabel_configs:
....
kubernetes_sd_configs:
- role: endpoints
kubeconfig_file: ""
follow_redirects: true
enable_http2: true
namespaces:
own_namespace: false
names:
- shop
Do I need to add any more elements? so that Service Monitor can collect metrics from other namespaces?
I already validated in my ClusterRole and Prometheus Operator has all permissions in namespaces.
Problem solved!
In Kube-prometheus by default it maps namespaces: (default, kube-system, monitoring)
To add specific namespaces we must edit the two files: prometheus-roleSpecificNamespaces.yaml
- apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
labels:
app.kubernetes.io/component: prometheus
app.kubernetes.io/instance: k8s
app.kubernetes.io/name: prometheus
app.kubernetes.io/part-of: kube-prometheus
app.kubernetes.io/version: 2.38.0
name: prometheus-k8s
namespace: loja
rules:
- apiGroups:
- ""
resources:
- services
- endpoints
- pods
verbs:
- get
- list
- watch
- apiGroups:
- extensions
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- networking.k8s.io
resources:
- ingresses
verbs:
- get
- list
- watch- apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
labels:
app.kubernetes.io/component: prometheus
app.kubernetes.io/instance: k8s
app.kubernetes.io/name: prometheus
app.kubernetes.io/part-of: kube-prometheus
app.kubernetes.io/version: 2.38.0
name: prometheus-k8s
namespace: loja
rules:
- apiGroups:
- ""
resources:
- services
- endpoints
- pods
verbs:
- get
- list
- watch
- apiGroups:
- extensions
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- networking.k8s.io
resources:
- ingresses
verbs:
- get
- list
- watch
prometheus-roleBindingSpecificNamespaces.yaml
- apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
app.kubernetes.io/component: prometheus
app.kubernetes.io/instance: k8s
app.kubernetes.io/name: prometheus
app.kubernetes.io/part-of: kube-prometheus
app.kubernetes.io/version: 2.38.0
name: prometheus-k8s
namespace: loja
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: prometheus-k8s
subjects:
- kind: ServiceAccount
name: prometheus-k8s
namespace: monitoring
After adding the desired namespace and applying the yml, the ServiceMonitor will be added.