We deployed the metricbeat on our Kubernetes cluster. All the metrics are been pushed to the elasticsearch index without any issues. But observing below error in the metribeat logs and not sure what exactly is causing this.
2020-07-01T14:05:25.421Z INFO module/wrapper.go:259 Error fetching data for metricset kubernetes.apiserver: error getting metrics: unexpected status code 403 from server
We have explored and tried the suggested cluster-role and didn't see any success. Below is the cluster-role we are using.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: metricbeat
labels:
k8s-app: metricbeat
rules:
- apiGroups:
- "extensions"
- "apps"
- "v1"
- ""
resources:
- namespaces
- pods
- events
- deployments
- nodes
- replicasets
- statefulsets
- nodes/metrics
- nodes/stats
verbs:
- get
- list
- watch
Let me know if anyone has came across the same and have fixed it. Thanks in advance.
It looks like Metricbeat cannot connect to the K8S API server, HTTP 403 is a security error.
According to the official documentation, can you make sure that you add these in the ClusterRole rules:
rules:
- nonResourceURLs:
- /metrics
verbs:
- get