kuberneteskubernetes-apiserverkube-apiserver

How Do I Properly Set --kubelet-certificate-authority apiserver parameter?

I am using KubeSpray to provision a two node cluster on AWS. By default, the --kubelet-certificate-authority parameter is not used. However, I would like to set it.

I do not know the correct setting for --kubelet-certificate-authority. When I set it to /etc/kubernetes/pki/ca.crt I see messages like the following in my logs:

TLS handshake error from 10.245.207.223:59278: remote error: tls: unknown certificate authority

In order to create an isolated environment for testing, I SSH'ed to my controller node to run this command which runs apiserver on the non-standard port of 4667. I copied these values directly from /etc/kubernetes/manifests/kube-apiserver.yaml. You'll need to adjust the value to match your own cluster. I purposely am running the container in interactive mode so that I can see the log messages.

sudo docker run \
  --name api-server-playground \
  -it \
  --rm \
  --network host \
  --volume /etc/kubernetes:/etc/kubernetes:ro \
  --volume /etc/pki:/etc/pki:ro \
  --volume /etc/ssl:/etc/ssl/:ro \
  k8s.gcr.io/kube-apiserver:v1.18.9 \
  kube-apiserver \
    --advertise-address=10.245.207.223 \
    --allow-privileged=true \
    --anonymous-auth=True \
    --apiserver-count=1 \
    --authorization-mode=Node,RBAC \
    --bind-address=0.0.0.0 \
    --client-ca-file=/etc/kubernetes/ssl/ca.crt \
    --cloud-config=/etc/kubernetes/cloud_config \
    --cloud-provider=aws \
    --enable-admission-plugins=NodeRestriction \
    --enable-aggregator-routing=False \
    --enable-bootstrap-token-auth=true \
    --endpoint-reconciler-type=lease \
    --etcd-cafile=/etc/ssl/etcd/ssl/ca.pem \
    --etcd-certfile=/etc/ssl/etcd/ssl/node-ip-10-245-207-223.ec2.internal.pem \
    --etcd-keyfile=/etc/ssl/etcd/ssl/node-ip-10-245-207-223.ec2.internal-key.pem \
    --etcd-servers=https://10.245.207.119:2379 \
    --event-ttl=1h0m0s \
    --insecure-port=0 \
    --kubelet-client-certificate=/etc/kubernetes/ssl/apiserver-kubelet-client.crt \
    --kubelet-client-key=/etc/kubernetes/ssl/apiserver-kubelet-client.key \
    --kubelet-preferred-address-types=InternalDNS,InternalIP,Hostname,ExternalDNS,ExternalIP \
    --profiling=False \
    --proxy-client-cert-file=/etc/kubernetes/ssl/front-proxy-client.crt \
    --proxy-client-key-file=/etc/kubernetes/ssl/front-proxy-client.key \
    --request-timeout=1m0s \
    --requestheader-allowed-names=front-proxy-client \
    --requestheader-client-ca-file=/etc/kubernetes/ssl/front-proxy-ca.crt \
    --requestheader-extra-headers-prefix=X-Remote-Extra- \
    --requestheader-group-headers=X-Remote-Group \
    --requestheader-username-headers=X-Remote-User \
    --secure-port=6447 \
    --service-account-key-file=/etc/kubernetes/ssl/sa.pub \
    --service-cluster-ip-range=10.233.0.0/18 \
    --service-node-port-range=30000-32767 \
    --storage-backend=etcd3 \
    --tls-cert-file=/etc/kubernetes/ssl/apiserver.crt \
    --tls-private-key-file=/etc/kubernetes/ssl/apiserver.key

Now it's possible to SSH into the controller again and use curl to interact with the custom apiserver container.

APISERVER=https://10.245.207.223:6447

TOKEN=$(kubectl get secrets -o jsonpath="{.items[?(@.metadata.annotations['kubernetes\.io/service-account\.name']=='default')].data.token}"|base64 --decode); echo "TOKEN=$TOKEN"

curl --header "Authorization: Bearer $TOKEN" -X GET $APISERVER/api

An error message will appear in the docker log. It will look like this:

I0921 14:39:07.662368       1 log.go:172] http: TLS handshake error 
from 10.245.207.223:59278: remote error: tls: unknown certificate authority

curl -k --header "Authorization: Bearer $TOKEN" -X GET $APISERVER/api
{
  "kind": "APIVersions",
  "versions": [
    "v1"
  ],
  "serverAddressByClientCIDRs": [
    {
      "clientCIDR": "0.0.0.0/0",
      "serverAddress": "10.245.207.223:6447"
    }
  ]
}

You'll see the request works correctly. However, I don't want to use the -k parameter. So I tried to use the certificate authority from the apiserver.

echo | \
    openssl s_client -connect $APISERVER 2>/dev/null | \
    openssl x509 -text | \
    sed -n "/BEGIN CERTIFICATE/,/END CERTIFICATE/p" \
    > apiserver.ca.crt

curl --cacert apiserver.ca.crt --header "Authorization: Bearer $TOKEN" -X GET $APISERVER/api

UPDATE

Following a thought prompted by Wiktor's response, I added /etc/kubernetes/ssl/ca.crt as the certificate authority. And used that file in my curl command.

curl --cacert /etc/kubernetes/ssl/ca.crt --header "Authorization: Bearer $TOKEN" -X GET $APISERVER/api

This worked.

Solution

  • In order to make the --kubelet-certificate-authority flag work you first need to make sure you got Kubelet authentication and Kubelet authorization enabled. After that you can follow the Kubernetes documentation and setup the TLS connection between the apiserver and kubelet. And finally, you can edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the master node and set the --kubelet-certificate-authority parameter to the path to the cert file for the certificate authority.

    So, to sum up the steps to do are:

    1. Kubelet authentication:
    1. Kubelet authorization:
    1. Use the --kubelet-certificate-authority flag to provide the apiserver with a root certificate bundle to use to verify the kubelet's serving certificate.

    More details can be found in the linked documentation.