kuberneteskubernetes-helmfluxcd

helm error: release requires a rollback before it can be upgraded

In my cluster Im using weave flux along with their flux-helm-operator to manage my cluster the gitops way.

However, when I make a change to the chart at flux git repository, I often come across the following error msg:

ts=2019-09-25T11:54:37.604506452Z caller=chartsync.go:328 component=chartsync
warning="unable to proceed with release" 
resource=mychart:helmrelease/mychart release=mychart
err="release requires a rollback before it can be upgraded (FAILED)"

Im not sure what does it mean in helm, but anyway, I am not supposed to run any helm command since the releases are managed by flux so I am wondering what is the correct way to deal with this error on production

(besides deleting the release and waiting for flux to recreate it)

A well-explained answer will be very much accepted, Thanks.

Solution

  • Let's dive into the code of helm-operator

    Warning unable to proceed with release arises after GetUpgradableRelease

        // GetUpgradableRelease returns a release if the current state of it
    // allows an upgrade, a descriptive error if it is not allowed, or
    // nil if the release does not exist.

    It returns error release requires a rollback before it can be upgraded if release has Status_FAILED state (see release.go#89 )

    UNHEALTHY state blocks release

    As flux developers mentioned in #2265, there is no way to roll to UNHEALTHY state.

    This is not a bug but I can see where your expectation is coming from.

    Flux will only move healthy releases forward, one of the reasons for this is to ensure we do not end up in a loop of failure, the --force flag is thus not intended to be used to force the upgrade of an unhealthy resource (you should use the rollback feature for this) but was developed to make it possible to upgrade charts with e.g. backwards incompatible changes (changes on immutable fields for example, which require a resource to be removed first, see #1760).

    Conclusion: the forceUpgrade is honoured, but can not be used to force the upgrade of a release in an UNHEALTHY state.

    Rollback

    As author recommends, you should use rollback feature

    From time to time a release made by the Helm operator may fail, it is possible to automate the rollback of a failed release by setting .spec.rollback.enable to true on the HelmRelease resource.

    Note: a successful rollback of a Helm chart containing a StatefulSet resource is known to be tricky, and one of the main reasons automated rollbacks are not enabled by default for all HelmReleases. Verify a manual rollback of your Helm chart does not cause any problems before enabling it.

    When enabled, the Helm operator will detect a faulty upgrade and perform a rollback, it will not attempt a new upgrade unless it detects a change in values and/or the chart.

    apiVersion: flux.weave.works/v1beta1
kind: HelmRelease
# metadata: ...
spec:
  # Listed values are the defaults.
  rollback:
    # If set, will perform rollbacks for this release.
    enable: false
    # If set, will force resource update through delete/recreate if
    # needed.
    force: false
    # Prevent hooks from running during rollback.
    disableHooks: false
    # Time in seconds to wait for any individual Kubernetes operation.
    timeout: 300
    # If set, will wait until all Pods, PVCs, Services, and minimum
    # number of Pods of a Deployment are in a ready state before
    # marking the release as successful. It will wait for as long
    # as the set timeout.
    wait: false