helm upgrade fail after upgrade to Kubernetes 1.25 due to change in HorizontalPodAutoscaler API version

After upgrading Kubernetes to 1.25

helm upgrade --install ... fails with the following error:

Error: UPGRADE FAILED: unable to recognize "": no matches for kind "HorizontalPodAutoscaler" in version "autoscaling/v2beta1"

In order to resolve it I changed the HPA from autoscaling/v2beta1 to autoscaling/v2 and update the new API syntax. but I keep getting the same error when trying to upgrade the helm release. The only way to resolve it was to uninstall and reinstall the release. can someone explain the reason for the error and how to fix it without deleting and reinstall it?

Solution

helm3 keeps the release state in secret, the last release helm state contains the old API autoscaling/v2beta1, and for some reason, it cause an error on the upgrade. To resolve it, I edit the helm secret unpack the .data.release with base64 encode twice, and unzip replace autoscaling/v2beta1 with autoscaling/v2 then zip it encode twice.

After this change and the change for the new API version (and syntax), the problem was solved, and I could upgrade the chart again.

My fix:

Update the hpa template to the latest API version
Update (patch) the last helm secret (secret/sh.helm.release.v1....) with the following commands:

UPDATE=$(kubectl get secret "${SECRET}" -n "${NAMESPACE}" -otemplate='{{.data.release |base64decode |base64decode }}'|gzip -d|sed 's#autoscaling/v2beta1#autoscaling/v2#'| gzip |base64 -w0 |base64 -w0)
kubectl patch secret "${SECRET}" -n "${NAMESPACE}" --patch="{\"data\": { \"release\": \"$UPDATE\" }}"

I used this script for updating all helm secrets in all my namespaces.

#!/bin/bash


SECRET=$1
NAMESPACE=$2

if [[ -z "${SECRET}" ]]
then
    echo "Usage: $0 <secret-name> <namespace>"
    exit
fi

if [[ -z "${NAMESPACE}" ]]
then
    echo "Usage: $0 <secret-name> <namespace>"
    exit
fi

UPDATE=$(kubectl get secret "${SECRET}" -n "${NAMESPACE}" -otemplate='{{.data.release |base64decode |base64decode }}'|gzip -d|sed 's#autoscaling/v2beta1#autoscaling/v2#'| gzip |base64 -w0 |base64 -w0)
kubectl patch secret "${SECRET}" -n "${NAMESPACE}" --patch="{\"data\": { \"release\": \"$UPDATE\" }}"


# Running example:
## Fix single secret
# ./fix-helm-hpa.sh <secret> <namespace>

## Fix all secrets
#kubectl get secret --field-selector=type=helm.sh/release.v1 -otemplate='{{range .items}}{{printf "%s %s\n" .metadata.name .metadata.namespace }}{{end}}' |while read line ; do ./fix-helm-hpa.sh $line; done