How to convert a Kubernetes non-HA control plane into an HA control plane?

What is the best way to convert a kubernetes non-HA control plane into an HA control plane?

I have started with my cluster as a non-HA control plane - one master node and several worker nodes. The cluster is already running with a lots of services.

Now I would like to add additional master nodes to convert my cluster into a HA control plane. I have setup and configured a load balancer. But I did not figure out how I can change the -control-plane-endpoint to my load balancer IP address for my existing master node.

Calling kubeadm results in the following error:

sudo kubeadm init --control-plane-endpoint "my-load-balancer:6443" --upload-certs
[init] Using Kubernetes version: v1.20.1
[preflight] Running pre-flight checks
    [WARNING SystemVerification]: missing optional cgroups: hugetlb
error execution phase preflight: [preflight] Some fatal errors occurred:
    [ERROR Port-6443]: Port 6443 is in use
    [ERROR Port-10259]: Port 10259 is in use
    [ERROR Port-10257]: Port 10257 is in use
    [ERROR FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml]: /etc/kubernetes/manifests/kube-apiserver.yaml already exists
    [ERROR FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml]: /etc/kubernetes/manifests/kube-controller-manager.yaml already exists
    [ERROR FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml]: /etc/kubernetes/manifests/kube-scheduler.yaml already exists
    [ERROR FileAvailable--etc-kubernetes-manifests-etcd.yaml]: /etc/kubernetes/manifests/etcd.yaml already exists
    [ERROR Port-10250]: Port 10250 is in use
    [ERROR Port-2379]: Port 2379 is in use
    [ERROR Port-2380]: Port 2380 is in use
    [ERROR DirAvailable--var-lib-etcd]: /var/lib/etcd is not empty
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher

The error message seems to be clear as my master is already running. Is there a way how I can easily tell my existing master node to use the new load balancer to run as a HA control plane?

Solution

Best solution in my opinion

The best approach to convert a non-HA control plane to an HA control plane is to create a completely new HA control plane and after that to migrate all your applications there.

Possible solution

Below I will try to help you to achieve your goal but I do not recommend using this procedure on any cluster that will ever be considered as production cluster. It work for my scenario, it also might help you.

Update the kube-apiserver certificate

First of all, kube-apiserver uses a certificate to encrypt control plane traffic and this certificate have something known as SAN (Subject Alternative Name). SAN is a list of IP addresses that you will use to access the API, so you need to add there IP address of your LoadBalancer and probably the hostname of your LB as well.

To do that, you have to get kubeadm configuration e.g. using command:

   $ kubeadm config view > kubeadm-config.yaml

and then add certSANs to kubeadm-config.yaml config file under apiServer section, it should looks like below example: (you may also need to add controlPlaneEndpoint to point to your LB).

apiServer:
  certSANs:
  - "192.168.0.2" # your LB address
  - "loadbalancer" # your LB hostname
  extraArgs:
    authorization-mode: Node,RBAC
    ...
 controlPlaneEndpoint: "loadbalancer" # your LB DNS name or DNS CNAME
    ...

Now you can update kube-apiserver cert using:
BUT please remember you must first delete/move your old kube-apiserver cert and key from /etc/kubernetes/pki/ !

$ kubeadm init phase certs apiserver --config kubeadm-config.yaml.

Finally restart your kube-apiserver.

Update the kubelet, the scheduler and the controller manager kubeconfig files

Next step is to update the kubelet, scheduler and controller manager to communicate with the kube-apiserver using LoadBalancer.
All three of these components use standard kubeconfig files:
/etc/kubernetes/kubelet.conf, /etc/kubernetes/scheduler.conf, /etc/kubernetes/controller-manager.conf to communicate with kube-apiserver.
The only thing to do is to edit the server: line to point to LB instead of kube-apiserver directly and then restart these components.

The kubelet is systemd service so to restart it use:

systemctl restart kubelet

the controller manager and schedulers are deployed as pods.

Update the kube-proxy kubeconfig files

Next it is time to update kubeconfig file for kube-proxy and same as before - the only thing to do is to edit the server: line to point to LoadBalancer instead of kube-apiserver directly.

This kubeconfig is in fact a configmap, so you can edit it directly using:

$ kubectl edit cm kube-proxy -n kube-system

or first save it as manifest file:

$ kubectl get cm kube-proxy -n kube-system -o yaml > kube-proxy.yml

and then apply changes.

Don't forget to restart kube-proxy after these changes.

Update the kubeadm-config configmap

At the end upload new kubeadm-config configmap (with certSANs and controlPlaneEndpoint entries) to the cluster, it's especially important when you want to add new node to the cluster.

$ kubeadm config upload from-file --config kubeadm-config.yaml

if command above doesn't work, try this:

$ kubeadm upgrade apply --config kubeadm-config.yaml