[SOLVED] Kubernetes Dashboard - Unknown server error after login

I´ve successfully deployed Kubernetes via Kubespray and everything seems to work fine. I am able to access the cluster via kubectl and list nodes, pods, services, secrets and so on. It`s also possible to apply new ressources and dashboard endpoint gets me the dashboard login page.

I´ve logged in with tokens of different serviceaccounts (default, kubernetes-dashboard, kubernetes-admin, ...)...with every login I get the same popups as described in kubespray dashboard warning forbidden popups for example.

So I applied the clusterrolebinding for default service account as described. When I login now with the defaults account token, I only get a

Unknown Server Error (404)
the server could not find the requested resource
Redirecting to previous state in 3 seconds...

box which redirects me to the login page afterwards. Its the same behaviour if I connect to Dashboard via kubectl proxy. The access is HTTPS over a public cluster IP and also HTTP over proxy

I am using Kubernetes 1.16.2 and latest Kubespray master commit 18d19d9e

EDIT: I destroyed and reprovisioned the cluster to get a fresh Kubespray-provisioned instance to make all steps deterministic, adding more info...

kubectl -n kube-system logs --follow kubernetes-dashboard-556b9ff8f8-jbmgg -- during an login attempt gives me

2019/12/16 12:35:02 [2019-12-16T12:35:02Z] Incoming HTTP/2.0 GET /api/v1/csrftoken/login request from 10.233.74.0:57458: { contents hidden }
2019/12/16 12:35:02 [2019-12-16T12:35:02Z] Outcoming response to 10.233.74.0:57458 with 200 status code
2019/12/16 12:35:02 [2019-12-16T12:35:02Z] Incoming HTTP/2.0 POST /api/v1/login request from 10.233.74.0:57458: { contents hidden }
2019/12/16 12:35:02 [2019-12-16T12:35:02Z] Outcoming response to 10.233.74.0:57458 with 200 status code
2019/12/16 12:35:02 [2019-12-16T12:35:02Z] Incoming HTTP/2.0 GET /api/v1/login/status request from 10.233.74.0:57458: {}
2019/12/16 12:35:02 [2019-12-16T12:35:02Z] Outcoming response to 10.233.74.0:57458 with 200 status code
2019/12/16 12:35:02 [2019-12-16T12:35:02Z] Incoming HTTP/2.0 GET /api/v1/csrftoken/token request from 10.233.74.0:57458: {}
2019/12/16 12:35:02 [2019-12-16T12:35:02Z] Outcoming response to 10.233.74.0:57458 with 200 status code
2019/12/16 12:35:02 [2019-12-16T12:35:02Z] Incoming HTTP/2.0 POST /api/v1/token/refresh request from 10.233.74.0:57458: { contents hidden }
2019/12/16 12:35:02 [2019-12-16T12:35:02Z] Outcoming response to 10.233.74.0:57458 with 200 status code
2019/12/16 12:35:02 [2019-12-16T12:35:02Z] Incoming HTTP/2.0 GET /api/v1/login/status request from 10.233.74.0:57458: {}
2019/12/16 12:35:02 [2019-12-16T12:35:02Z] Outcoming response to 10.233.74.0:57458 with 200 status code
2019/12/16 12:35:02 [2019-12-16T12:35:02Z] Incoming HTTP/2.0 GET /api/v1/csrftoken/token request from 10.233.74.0:57458: {}
2019/12/16 12:35:02 [2019-12-16T12:35:02Z] Outcoming response to 10.233.74.0:57458 with 200 status code
2019/12/16 12:35:02 [2019-12-16T12:35:02Z] Incoming HTTP/2.0 POST /api/v1/token/refresh request from 10.233.74.0:57458: { contents hidden }
2019/12/16 12:35:02 [2019-12-16T12:35:02Z] Outcoming response to 10.233.74.0:57458 with 200 status code
2019/12/16 12:35:02 [2019-12-16T12:35:02Z] Incoming HTTP/2.0 GET /api/v1/overview/default?filterBy=&itemsPerPage=10&name=&page=1&sortBy=d,creationTimestamp request from 10.233.74.0:57458: {}
2019/12/16 12:35:03 Getting config category
2019/12/16 12:35:03 Getting discovery and load balancing category
2019/12/16 12:35:03 Getting lists of all workloads
2019/12/16 12:35:03 the server could not find the requested resource
2019/12/16 12:35:03 [2019-12-16T12:35:03Z] Outcoming response to 10.233.74.0:57458 with 404 status code
2019/12/16 12:35:03 No metric client provided. Skipping metrics.
2019/12/16 12:35:03 No metric client provided. Skipping metrics.
2019/12/16 12:35:03 No metric client provided. Skipping metrics.
2019/12/16 12:35:03 Getting pod metrics
2019/12/16 12:35:03 No metric client provided. Skipping metrics.
2019/12/16 12:35:03 No metric client provided. Skipping metrics.
2019/12/16 12:35:03 [2019-12-16T12:35:03Z] Incoming HTTP/2.0 GET /api/v1/systembanner request from 10.233.74.0:57458: {}
2019/12/16 12:35:03 [2019-12-16T12:35:03Z] Incoming HTTP/2.0 GET /api/v1/login/status request from 10.233.74.0:57458: {}
2019/12/16 12:35:03 [2019-12-16T12:35:03Z] Incoming HTTP/2.0 GET /api/v1/rbac/status request from 10.233.74.0:57458: {}
2019/12/16 12:35:03 [2019-12-16T12:35:03Z] Outcoming response to 10.233.74.0:57458 with 200 status code
2019/12/16 12:35:03 [2019-12-16T12:35:03Z] Outcoming response to 10.233.74.0:57458 with 200 status code
2019/12/16 12:35:03 [2019-12-16T12:35:03Z] Outcoming response to 10.233.74.0:57458 with 200 status code
2019/12/16 12:35:12 Metric client health check failed: the server could not find the requested resource (get services heapster). Retrying in 30 seconds.
2019/12/16 12:35:42 Metric client health check failed: the server could not find the requested resource (get services heapster). Retrying in 30 seconds.

I`ve found a strange workaround to get dashboard working, but this is not usable for us in production, perhaps somebody can explain this:

I take for example the serviceaccount kube-system:default (Note: this one is NOT assigned cluster-admin at this point
I get its token and login with that
Dashboard obviously shows me the "forbidden-popups"
While still logged in, I run kubectl create clusterrolebinding default-admin --clusterrole cluster-admin --serviceaccount=kube-system:default
I refresh the browser tab which holds my dashboard session...et voila, everything shows up correctly.

Therefore I am not able to logout and login again, I always have to delete the clusterrolebinding, login then and afterwards apply clusterrolebinding again.

This seems to be strongly related to kubespray provisioned clusters, so anybody able to reproduce this with kubespray?