We're using Spring Boot 3.0.5 with leader election from Spring Cloud Kubernetes version 3.0.2 (org.springframework.cloud:spring-cloud-kubernetes-fabric8-leader:3.0.2) that is based on the fabric8's Java client.
However, after moving to a new Kubernetes cluster which is more restrictive by default (i.e. pods are not using the "default service account" that has access to everything), we can't get the leader election to work (it used to work when the pod had access to "everything" using the default service account). We've configured the following rules:
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
namespace: default
name: configmap-editor
rules:
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["get", "list", "watch", "create", "delete", "write"]
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
namespace: default
name: pod-viewer
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list", "watch"]
And bindings for the pod that uses the leader election:
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: configmap-editor-binding
namespace: default
subjects:
- kind: ServiceAccount
name: my-app
namespace: default
roleRef:
kind: Role
name: configmap-editor
apiGroup: rbac.authorization.k8s.io
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: pod-viewer-binding
namespace: default
subjects:
- kind: ServiceAccount
name: my-app
namespace: default
roleRef:
kind: Role
name: pod-viewer
apiGroup: rbac.authorization.k8s.io
The Kubernetes service account referenced in the RoleBinding, is then connected to a Google Cloud Service Account using workload identity. The only permission that it currently has, on the Google Cloud side, is "Cloud Trace Agent".
We don't get any error messages when booting our application, but our function that takes an OnGrantedEvent from Spring Cloud Kubernetes, is never received:
@EventListener
fun onLeaderGranted(event: OnGrantedEvent): Unit = ...
What permissions and/or RBAC rules are we missing?
The problem was that the "configmap-editor" role didn't have enough rules. The correct rules should be:
["get", "watch", "list", "create", "update", "patch", "delete"]