kuberneteskubernetes-podhorizontal-scaling

offset Kubernetes pod scaling by a second


I have an application that I want to scale in parallel but would love a way to have a brief pause between each new pod getting created.

I need this because during pod creation I update a file, and if the file isn't closed by the time the next pod tries to open it and update it, it will fail.

So I just need a 1-second buffer between pods opening and closing this file.

Right now if a scale happens and more than 1 new pod is added, they hit the file at the same time. So one pod will work, the other will fail, and I have to wait for the pod to timeout for k8s to kill it and recreate it, at which point it will work, but wastes a lot of valuable time during scaling events when I need new pods as quickly as possible to handle the load.

Not sure if I'm not wording my searches correctly, but not able to find this on the k8s website. Any ideas would be helpful.

(additional note, I know on StatefulSet it will scale 1 pod at a time by default, However, that method requires all pods to be healthy to continue to add that one node at a time, and if any pod becomes unhealthy it will stop scaling until all pods are healthy again. Which during high load situations, wouldn't be ideal)


Solution

  • You will need to do some work in order to achieve it.

    Few "hacks/solutions"

    1. Init Container

    2. Probe 'PostStart'

    apiVersion: v1
    kind: Pod
    metadata:
      name: lifecycle-demo
    spec:
      containers:
      - name: ...
        image: ...
        lifecycle:
          postStart:
            exec:
              command: [<get the current number of pods from the replica> 
                        <check to see if you need to restart a new pod> 
                        <scale a new pod if required>]
          
    

    3. CronJob