Kubespray: Can't run upgrade due to cordon failing on istiod

I am running Kubespray v.2.19.0 on 4 bare metal servers running Ubuntu 20.04. The 4 servers are setup as one master node and 3 worker nodes. It's a small cluster I run at my house to run my own compute and to "play"

I am trying to run a upgrade using:

ansible-playbook -kK -i inventory/mycluster/hosts.yaml upgrade-cluster.yml --become --become-user=root -e ignore_assert_errors=yes

But when it tries to cordon off the master node (server name is server) I get the following:

TASK [upgrade/pre-upgrade : Drain node] ****************************************
fatal: [server -> server]: FAILED! => {"attempts": 3, "changed": true, "cmd": ["/usr/local/bin/kubectl", "--kubeconfig", "/etc/kubernetes/admin.conf", "drain", "--force", "--ignore-daemonsets", "--grace-period", "300", "--timeout", "360s", "--delete-emptydir-data", "server"], "delta": "0:06:02.042964", "end": "2023-05-28 10:14:07.440240", "failed_when_result": true, "msg": "non-zero return code", "rc": 1, "start": "2023-05-28 10:08:05.397276", "stderr": "WARNING: ignoring DaemonSet-managed Pods: ingress-nginx/ingress-nginx-controller-rmdtj, kube-system/calico-node-rmdjg, kube-system/kube-proxy-fb6ff, kube-system/nodelocaldns-w5tzq\nerror when evicting pods/\"istiod-6b56cffbd9-thdjl\" -n \"istio-system\" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.\nerror when evicting pods/\"istiod-6b56cffbd9-thdjl\" -n \"istio-system\" (will retry after 5s):...

Is there some flag I can pass to the upgrade to tell it to ignore the distribution budget or to ignore those errors altogether?

Solution

Rather than trying to create a workaround I'd suggest to resolve the actual problem.

You have two options. You can either increase the replicas to 2. Or you can disable pod disruption budget.

Since you only run a home lab I'd take option two to not waste resources on a second pod that you probably don't need.

Not clear how you installed istio, but that's how you do it using an operator manifest:

apiVersion: install.istio.io/v1alpha1 
kind: IstioOperator 
spec: 
  profile: default # your profile
  tag: 1.17.2 # your version 
  values: 
    global: 
      defaultPodDisruptionBudget: 
        enabled: false

Save it to a file like config.yaml and install it with istioctl install -f config.yaml.