kuberneteskubernetes-helm

Stopping all pods in Kubernetes cluster before running database migration job


I deploy my App into the Kubernetes cluster using Helm. App works with database, so i have to run db migrations before installing new version of the app. I run migrations with Kubernetes Job object using Helm "pre-upgrade" hook.

The problem is when the migration job starts old version pods are still working with database. They can block objects in database and because of that migration job may fail.

So, i want somehow to automatically stop all the pods in cluster before migration job starts. Is there any way to do that using Kubernetes + Helm? Will appreciate all the answers.


Solution

  • There are two ways I can see that you can do this.

    First option is to scale down the pods before the deployment (for example, via Jenkins, CircleCI, GitLab CI, etc)

    kubectl scale --replicas=0 -n {namespace} {deployment-name}
    helm install .....
    

    The second option (which might be easier depending on how you want to maintain this going forward) is to add an additional pre-upgrade hook with a higher priority than the migrations hook so it runs before the upgrade job; and then use that do the kubectl scale down.