configure kubelet to use rkt instead of docker on coreos

I'm trying to deploy master node as described in https://coreos.com/kubernetes/docs/latest/deploy-master.html on CoreOS beta (1153.4.0).

I'm using the following kubeconfig yaml file:

 current-context: tuxin-coreos-context
 apiVersion: v1
 clusters:
 - cluster:
     certificate-authority: /etc/kubernetes/ssl/ca.pem
     server: https://coreos-2.tux-in.com:443
   name: tuxin-coreos-cluster
 contexts:
 - context:
     cluster: tuxin-coreos-cluster
   name: tuxin-coreos-context
 kind: Config
 preferences:
   colors: true
 users:
 - name: ufk
   user:
     client-certificate: /etc/kubernetes/ssl/apiserver.pem
     client-key: /etc/kubernetes/ssl/apiserver-key.pem

I'm using KUBELET_VERSION v1.4.0-beta.10_coreos.0

systemctl cat kubelet returns the following:

/etc/systemd/system/kubelet.service

[Service]
ExecStartPre=/usr/bin/mkdir -p /etc/kubernetes/manifests
ExecStartPre=/usr/bin/mkdir -p /var/log/containers

Environment=KUBELET_VERSION=v1.4.0-beta.10_coreos.0
Environment="RKT_OPTS=--volume var-log,kind=host,source=/var/log \
  --mount volume=var-log,target=/var/log \
  --volume dns,kind=host,source=/etc/resolv.conf \
  --mount volume=dns,target=/etc/resolv.conf"

ExecStart=/usr/lib/coreos/kubelet-wrapper \
  --kubeconfig=/var/lib/kubelet/kubeconfig  \
  --network-plugin-dir=/etc/kubernetes/cni/net.d   \
  --network-plugin=cni   \
  --register-schedulable=false   \
  --allow-privileged=true  \
  --pod-manifest-path=/etc/kubernetes/manifests   \
  --hostname-override=coreos-2.tux-in.com   \
  --container-runtime=rkt
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target

journalctl -b -u kubelet returns

Sep 26 17:58:53 coreos-2.tux-in.com systemd[1]: Starting kubelet.service...
Sep 26 17:58:53 coreos-2.tux-in.com systemd[1]: Started kubelet.service.
Sep 26 17:58:53 coreos-2.tux-in.com kubelet-wrapper[1725]: image: using image from file /usr/lib64/rkt/stage1-images/stage1-fly.aci
Sep 26 17:58:54 coreos-2.tux-in.com kubelet-wrapper[1725]: image: using image from local store for image name quay.io/coreos/hyperkube:v1.4.0-beta.10_coreos.0
Sep 26 17:59:11 coreos-2.tux-in.com kubelet-wrapper[1725]: W0926 17:59:11.807917    1725 server.go:383] No API client: no api servers specified
Sep 26 17:59:12 coreos-2.tux-in.com kubelet-wrapper[1725]: I0926 17:59:12.871488    1725 manager.go:140] cAdvisor running in container: "/system.slice/kubelet.service"
Sep 26 17:59:14 coreos-2.tux-in.com kubelet-wrapper[1725]: I0926 17:59:14.454499    1725 fs.go:116] Filesystem partitions: map[/dev/sda9:{mountpoint:/var/lib/docker major:8 minor:9 fsType:ext4 blockSize:0} /dev/sda3:{mountpoint:/etc/ssl/certs major:8 minor:3 fsType:ext4 blockSize:0} overlay:{mountpoint:/ major:0 minor:54 fsType:overlay blockSize:0}]
Sep 26 17:59:14 coreos-2.tux-in.com kubelet-wrapper[1725]: I0926 17:59:14.458559    1725 manager.go:195] Machine: {NumCores:8 CpuFrequency:3060000 MemoryCapacity:4145504256 MachineID:f3e41f31b6024d84806ddf3a54075add SystemUUID:00000000-0000-0000-0000-1C6F653E6F72 BootID:ca4acb17-baa6-4863-acdb-2db06609b0d8 Filesystems:[{Device:/dev/sda3 Capacity:1031946240 Type:vfs Inodes:260096 HasInodes:true} {Device:overlay Capacity:984373800960 Type:vfs Inodes:61054976 HasInodes:true} {Device:/dev/sda9 Capacity:113819422720 Type:vfs Inodes:28536576 HasInodes:true}] DiskMap:map[8:0:{Name:sda Major:8 Minor:0 Size:120034123776 Scheduler:cfq} 8:16:{Name:sdb Major:8 Minor:16 Size:1000204886016 Scheduler:cfq} 8:32:{Name:sdc Major:8 Minor:32 Size:3000592982016 Scheduler:cfq} 8:48:{Name:sdd Major:8 Minor:48 Size:2000398934016 Scheduler:cfq} 8:64:{Name:sde Major:8 Minor:64 Size:1000204886016 Scheduler:cfq}] NetworkDevices:[{Name:enp4s0 MacAddress:1c:6f:65:3e:6f:72 Speed:1000 Mtu:1500} {Name:flannel.1 MacAddress:6a:5e:8b:1d:00:17 Speed:0 Mtu:1450}] Topology:[{Id:0 Memory:4145504256 Cores:[{Id:0 Threads:[0 4] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2}]} {Id:1 Threads:[1 5] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2}]} {Id:2 Threads:[2 6] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2}]} {Id:3 Threads:[3 7] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2}]}] Caches:[{Size:8388608 Type:Unified Level:3}]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 26 17:59:14 coreos-2.tux-in.com kubelet-wrapper[1725]: I0926 17:59:14.459359    1725 manager.go:201] Version: {KernelVersion:4.7.1-coreos ContainerOsVersion:CoreOS 1153.4.0 (MoreOS) DockerVersion:1.11.2 CadvisorVersion: CadvisorRevision:}
Sep 26 17:59:14 coreos-2.tux-in.com kubelet-wrapper[1725]: W0926 17:59:14.460736    1725 server.go:613] No api server defined - no events will be sent to API server.
Sep 26 17:59:14 coreos-2.tux-in.com kubelet-wrapper[1725]: I0926 17:59:14.460794    1725 kubelet.go:255] Adding manifest file: /etc/kubernetes/manifests
Sep 26 17:59:14 coreos-2.tux-in.com kubelet-wrapper[1725]: W0926 17:59:14.463983    1725 kubelet_network.go:64] Hairpin mode set to "promiscuous-bridge" but container runtime is "rkt", ignoring
Sep 26 17:59:14 coreos-2.tux-in.com kubelet-wrapper[1725]: I0926 17:59:14.464034    1725 kubelet.go:516] Hairpin mode set to "none"
Sep 26 17:59:14 coreos-2.tux-in.com kubelet-wrapper[1725]: Error: failed to run Kubelet: failed to create kubelet: cannot find rkt binary: exec: "rkt": executable file not found in $PATH
Sep 26 17:59:14 coreos-2.tux-in.com kubelet-wrapper[1725]: failed to run Kubelet: failed to create kubelet: cannot find rkt binary: exec: "rkt": executable file not found in $PATH
Sep 26 17:59:14 coreos-2.tux-in.com systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Sep 26 17:59:14 coreos-2.tux-in.com systemd[1]: kubelet.service: Unit entered failed state.
Sep 26 17:59:14 coreos-2.tux-in.com systemd[1]: kubelet.service: Failed with result 'exit-code'.
Sep 26 17:59:24 coreos-2.tux-in.com systemd[1]: kubelet.service: Service hold-off time over, scheduling restart.
Sep 26 17:59:24 coreos-2.tux-in.com systemd[1]: Stopped kubelet.service.

I tried to specify specific path to rkt, I tried to the same path that I have it on my core os which is /bin/rkt but it can't find the binary there. is the hypercube image is missing the rkt binary for some reason?

update

ok I tried to debug this issue further but got to a dead end again.

I tried to run everything manually to see where's the problem..

so I set the following environment variables:

export KUBELET_VERSION=v1.4.0_coreos.0
export RKT_OPTS=--volume var-log,kind=host,source=/var/log \
 --mount volume=var-log,target=/var/log \
 --volume dns,kind=host,source=/etc/resolv.conf \
 --mount volume=dns,target=/etc/resolv.conf

then I copied /usr/lib/coreos/kubelet-wrapper file to my home directory and modified it to run bash instead:

...
${KUBELET_ACI}:${KUBELET_VERSION} --exec=/bin/bash

then I ran kubelet with the following parameters:

./kubelet-wrapper   --kubeconfig=/var/lib/kubelet/kubeconfig    --network-plugin-dir=/etc/kubernetes/cni/net.d     --network-plugin=cni     --register-schedulable=false     --allow-privileged=true    --pod-manifest-path=/etc/kubernetes/manifests     --hostname-override=coreos-2.tux-in.com     --container-runtime=rkt

then after getting a bash prompt, I ran kubelet with the following parameters:

./kubelet --kubeconfig=/var/lib/kubelet/kubeconfig  \
  --network-plugin-dir=/etc/kubernetes/cni/net.d   \
  --network-plugin=cni   \
  --register-schedulable=false   \
  --allow-privileged=true  \
  --pod-manifest-path=/etc/kubernetes/manifests   \
  --hostname-override=coreos-2.tux-in.com   \
  --container-runtime=rkt

and I got again the cannot find rkt binary error message. I tried finding rkt binary in that container by running find / | grep rkt$ but no rkt binary found.

Solution

The easiest way to setup kubernetes with rkt as the runtime is to use the scripts found here. As Rob stated, the reason you see that error is because the kubelet is being run inside a container and doesn't see the /usr/bin/rkt on the host.

Right now, getting the containerized kubelet to properly talk to the rkt runtime on the host is a little complicated. Just just bind-mounting the rkt binary into the kubelet container is not enough. The details are captured in this PR.

We've spent some effort making this all work smoothly in the coreos-kubernetes repo and testing the result. If you use the scripts in that repository things should just work by setting the CONTAINER_RUNTIME environment variable to rkt.