[SOLVED] RBAC access denied when deploying an API on kubeflow cluster

RBAC access denied when deploying an API on kubeflow cluster

I have deployed kubeflow on a cluster and I managed to configure everything to access it properly.

The problem is when I tried deploying a small API (Deployment & LoadBalancer) on that same cluster in a different namespace and accessing it.

My user should have all the rights and I manage to deploy it properly but when I try to access it on the default route I'm getting RBAC: access denied

Solution

Try Below Solutions :

1) If a pod doesn't have sidecar.istio.io/inject annotation, it will not be injected Sidecar by default. And because of the istio-system's global-deny-all AuthorizationPolicy policy, the pod is not accessible.

$ kubectl get AuthorizationPolicy -A
NAMESPACE                   NAME                              AGE
istio-system                cluster-local-gateway             5d10h
istio-system                global-deny-all                   5d10h
istio-system                istio-ingressgateway              5d10h
...

$ kubectl get AuthorizationPolicy global-deny-all -n istio-system -o yaml 
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"security.istio.io/v1beta1","kind":"AuthorizationPolicy","metadata":{"annotations":{},"name":"global-deny-all","namespace":"istio-system"},"spec":{}}
  creationTimestamp: "2021-06-11T14:51:22Z"
  generation: 1
  name: global-deny-all
  namespace: istio-system
  resourceVersion: "1191"
  uid: e8b243c6-23c9-48a7-8947-b297ffd7d302
spec: {}

Set an allow-all AuthorizationPolicy, the serving pod was accessible

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: allow-all
  namespace: kubeflow-user-example-com
spec:
 rules:
 - {}

Note : You should not apply this AuthorizationPolicy, as it allows all traffic from anywhere to access your namespace. Which would also mean others can access your notebook instances, for example.

You do want to allow all communication within the namespace, so you try to add a selector for the same namespace.

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: allow-all-from-same-ns
  namespace: kubeflow-user-example-com
spec:
  action: ALLOW
  rules:
  - from:
     - source:
        namespaces: ["kubeflow-user-example-com"]

2) Disable Sidecar for the namespace using a kubectl command. Another option would be to disable Istio injection for the xxx pods by adding the sidecar.istio.io/inject: "false" label to the pods.

You can find how to disable Istio Sidecar by adding a label for the namespace. Disable the Sidecar for the two related pods by running the commands below.

$ kubectl label pods <20210601-zjwzh-3877683498> -n kubeflow-user-example-com sidecar.istio.io/inject="false"  --overwrite
 
$ k label pod <xxx-infraval-modelserver-72n7m> -n kubeflow-user-example-com sidecar.istio.io/inject="false"  --overwrite

3) Update the ServiceRoleBinding which is actually generated/monitored/managed by the profile controller in the kubeflow namespace instead of the validating webhook.

The RBAC issue is based on the params.yaml in the profiles manifest folder, the rule is generated as

request.headers[]: regal.l.c.lei@XXXX.com

instead of

request.headers[kubeflow-userid]: regal.l.c.lei@XXXX.com

Check you may mis-configed the value as blank instead of userid-header=kubeflow-userid in the params.yaml. Also please check the kubernetes official document how to Determine Whether a Request is Allowed or Denied for more information.