I'm trying to add my custom scheduler with scheduler plugin in Kubernetes. Due to no detailed example in Kubernetes document (scheduling-framework, 20180409-scheduling-framework)
So I simply made it for testing. But, I got error: unable to recognize "test.yaml": no matches for kind "KubeSchedulerConfiguration" in version "kubescheduler.config.k8s.io/v1alpha1".
How could I configure and use scheduler plugin?
In my case, test configurations are
apiVersion: kubescheduler.config.k8s.io/v1alpha1
kind: KubeSchedulerConfiguration
plugins:
queueSort:
enabled:
- name: test
pluginConfig:
- name: test-
kubectl command
$ kubectl apply -f test.yaml
error: unable to recognize "test.yaml": no matches for kind "KubeSchedulerConfiguration" in version "kubescheduler.config.k8s.io/v1alpha1"
When I created as pod,
$ kubectl create -f test.yaml
$ kubectl logs pod test
E0920 15:55:47.970707 1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "persistentvolumeclaims" in API group "" at the cluster scope
E0920 15:55:47.971808 1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "storageclasses" in API group "storage.k8s.io" at the cluster scope
E0920 15:55:48.962810 1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "nodes" in API group "" at the cluster scope
E0920 15:55:48.963476 1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1beta1.CSINode: csinodes.storage.k8s.io is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "csinodes" in API group "storage.k8s.io" at the cluster scope
E0920 15:55:48.964615 1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "statefulsets" in API group "apps" at the cluster scope
E0920 15:55:48.965674 1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "replicasets" in API group "apps" at the cluster scope
E0920 15:55:48.966800 1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "replicationcontrollers" in API group "" at the cluster scope
E0920 15:55:48.967967 1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.Service: services is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "services" in API group "" at the cluster scope
E0920 15:55:48.969030 1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "pods" in API group "" at the cluster scope
E0920 15:55:48.970095 1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "persistentvolumes" in API group "" at the cluster scope
E0920 15:55:48.971150 1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1beta1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "poddisruptionbudgets" in API group "policy" at the cluster scope
E0920 15:55:48.972252 1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "persistentvolumeclaims" in API group "" at the cluster scope
E0920 15:55:48.973396 1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "storageclasses" in API group "storage.k8s.io" at the cluster scope
E0920 15:55:49.964805 1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.Node: nodes is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "nodes" in API group "" at the cluster scope
E0920 15:55:49.965338 1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1beta1.CSINode: csinodes.storage.k8s.io is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "csinodes" in API group "storage.k8s.io" at the cluster scope
E0920 15:55:49.966313 1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "statefulsets" in API group "apps" at the cluster scope
E0920 15:55:49.967467 1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "replicasets" in API group "apps" at the cluster scope
E0920 15:55:49.968448 1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "replicationcontrollers" in API group "" at the cluster scope
E0920 15:55:49.969534 1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.Service: services is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "services" in API group "" at the cluster scope
E0920 15:55:49.970775 1 reflector.go:123] pkg/mod/k8s.io/client-go@v0.0.0-20190918160344-1fbdaa4c8d90/tools/cache/reflector.go:96: Failed to list *v1.Pod: pods is f^C
Enviroment
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T11:13:54Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T11:05:50Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
go
$ go version
go version go1.13 linux/amd64
go module
$ cat go.sum
...
k8s.io/kubernetes v1.16.0 h1:WPaqle2JWogVzLxhN6IK67u62IHKKrtYF7MS4FVR4/E=
k8s.io/kube-scheduler v0.0.0-20190918162820-3b5c1246eb18 h1:WnbesbtFrWlZ8wwgDuP1KN+6E+XLDGzG4ztojMxfHsc=
...
The scheduler configuration is not applied to the API server with kubectl apply command, it is passed to the scheduler process with kube-scheduler --config=/path/to/config. Read more about it on Github.
--config is the entry where you should specify which parameters the scheduler will take. From the API’s perspective, that config file is supposed to contain a KubeSchedulerConfiguration object:
# content of the file passed to "--config"
apiVersion: kubescheduler.config.k8s.io/v1alpha1
kind: KubeSchedulerConfiguration
clientConnection:
kubeconfig: "/var/run/kubernetes/scheduler.kubeconfig"
algorithmSource:
policy:
file:
path: "/root/config/scheduler-extender-policy.json"
Refer to this article, which describe how to create custom Kubernetes scheduler and configuration of Scheduler.