I want to run a pod that listens for updates to endpoint lists (I'm not yet ready to adopt the alpha-level feature of endpoint sets, but I'll expand to that eventually.)
I have this code:
package main
import (
"fmt"
"os"
"os/signal"
"sync"
"syscall"
"k8s.io/apimachinery/pkg/util/runtime"
"k8s.io/client-go/informers"
"k8s.io/client-go/kubernetes"
"k8s.io/client-go/rest"
"k8s.io/client-go/tools/cache"
)
func ReadKubeConfig() (*rest.Config, *kubernetes.Clientset, error) {
config, err := rest.InClusterConfig()
if err != nil {
return nil, nil, err
}
clients, err := kubernetes.NewForConfig(config)
if err != nil {
return nil, nil, err
}
return config, clients, nil
}
func main() {
_, cs, err := ReadKubeConfig()
if err != nil {
fmt.Printf("could not create Clientset: %s\n", err)
os.Exit(1)
}
factory := informers.NewSharedInformerFactory(cs, 0)
ifmr := factory.Core().V1().Endpoints().Informer()
stop := make(chan struct{})
ifmr.AddEventHandler(cache.ResourceEventHandlerFuncs{
AddFunc: func(next interface{}) {
fmt.Printf("AddFunc(%v)\n", next)
},
UpdateFunc: func(prev, next interface{}) {
fmt.Printf("UpdateFunc(%v, %v)\n", prev, next)
},
DeleteFunc: func(prev interface{}) {
fmt.Printf("DeleteFunc(%v)\n", prev)
},
})
wg := &sync.WaitGroup{}
wg.Add(1)
go func() {
defer runtime.HandleCrash()
ifmr.Run(stop)
wg.Done()
}()
ch := make(chan os.Signal, 1)
signal.Notify(ch, os.Interrupt)
signal.Notify(ch, os.Signal(syscall.SIGTERM))
signal.Notify(ch, os.Signal(syscall.SIGHUP))
sig := <-ch
fmt.Printf("Received signal %s\n", sig)
close(stop)
wg.Wait()
}
I get this error when deploying and running:
kubeendpointwatcher.go:55: Failed to list *v1.Endpoints: endpoints is forbidden: User "system:serviceaccount:eng:default" cannot list resource "endpoints" in API group "" at the cluster scope
I have the following role and role binding defined and deployed to the "eng" namespace:
watch_endpoints$ kubectl -n eng get role mesh-endpoint-read -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
creationTimestamp: "2021-07-08T19:59:20Z"
name: mesh-endpoint-read
namespace: eng
resourceVersion: "182975428"
selfLink: /apis/rbac.authorization.k8s.io/v1/namespaces/eng/roles/mesh-endpoint-read
uid: fcadcc2a-19d0-4d6e-bee1-78413f51b91b
rules:
- apiGroups:
- ""
resources:
- endpoints
verbs:
- get
- list
- watch
I have the following rolebinding:
watch_endpoints$ kubectl -n eng get rolebinding mesh-endpoint-read -o yaml | sed -e 's/^/ /g'
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
creationTimestamp: "2021-07-08T19:59:20Z"
name: mesh-endpoint-read
namespace: eng
resourceVersion: "182977845"
selfLink: /apis/rbac.authorization.k8s.io/v1/namespaces/eng/rolebindings/mesh-endpoint-read
uid: 705a3e50-2a73-47ed-aa62-0ea48f3493ee
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: mesh-endpoint-read
subjects:
- kind: ServiceAccount
name: default
namespace: default
You will note that I apply it to both the default
namespace and the eng
namespace serviceaccount named default
although the error message seems to indicate that it is indeed running in the default
serviceaccount in the eng
namespace.
I have previously used Role and RoleBinding and ServiceAccount objects that work as expected, so I don't understand why this doesn't work. What am I missing?
For testing/reproduction purposes, I run this program by doing kubectl cp
of a built binary (cgo off) into a container created with kubectl -n eng create deplpoy
with a vanilla ubuntu
image running /bin/sh -c sleep 999999999
, and then executing a /bin/bash shell in that pod-container.
You have created role
and rolebinding
for eng
namespace. However, as per the error message:
kubeendpointwatcher.go:55: Failed to list *v1.Endpoints: endpoints is forbidden: User "system:serviceaccount:eng:default" cannot list resource "endpoints" in API group "" at the cluster scope
you are doing query for endpoints at the "cluster
" scope. Either try to limit your query to eng namespace or use clusterrole
/clusterbindings
Error message provide hint(system:serviceaccount:eng:default
)that serviceaccount
running in eng
namespace, whose name is default
does not have permission to query ep
at cluster scope.
To validate this, you can run two curl
calls, first exec
into the pod using the same sa
and then run the following for eng
namespace and later try it on other namespaces.
curl -v --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" https://kubernetes.default.svc/api/v1/namespaces/default/pods