kubernetesinternal-load-balancer

How to directly mark pod as offline in kubernetes loadbalancer

I have a k8s loadbalancer in front of some pods. When I deploy updated pods a certain amount of traffic to the pods will timeout and fail - it basically seems like k8s will handle the pod update/deploy as expected, but the loadbalancer will be unaware of this and will just continue to send traffic to all pods, until it seems liveness monitors fail and only then stop sending traffic to the pods.

What I'd love to see is the following scenario:

  1. mark a pod offline in the loadbalancer, so no new traffic is sent there - but without having the pod restarted
  2. update/replace pod
  3. send traffic to the new pod when it responds 200 to readiness probe

This way I wouldn't get any timeouts. So, how to achieve this? Or is it just me having a bad config, as it's not working?

Edit:

I have both readiness and liveness probes set up. The config for these:

        readinessProbe:
          httpGet:
             path: /ready
             port: 8080
          periodSeconds: 1
          initialDelaySeconds: 5
        livenessProbe:
          httpGet:
             path: /alive
             port: 8080
          periodSeconds: 1
          failureThreshold: 2
          initialDelaySeconds: 40

However, at a surface level, this does not fix the problem: if the loadbalancer will only stop sending traffic after a probe fails, and k8s starts bringing down a pod without notifying the loadbalancer, then there will be failed requests. It doesn't matter how I tweak the values above - periodSeconds has a minimum value of 1 second.

I am considering experimenting with the readiness probe though - to see if I can take a pod offline without restarting it, before starting deploy. Not exactly optimal, but maybe a way forward.

Solution

  • As @Daniel Lee mentioned in comments

    To achieve that what you want, you should configure health checks in your deployment.

    So let's start with what are the health checks?

    As mentioned here.

    Types of health checks

    Kubernetes gives you two types of health checks, and it is important to understand the differences between the two, and their uses.

    Readiness

    Readiness probes are designed to let Kubernetes know when your app is ready to serve traffic. Kubernetes makes sure the readiness probe passes before allowing a service to send traffic to the pod. If a readiness probe starts to fail, Kubernetes stops sending traffic to the pod until it passes.

    Liveness

    Liveness probes let Kubernetes know if your app is alive or dead. If you app is alive, then Kubernetes leaves it alone. If your app is dead, Kubernetes removes the Pod and starts a new one to replace it.

    Additionally, on above website you will find a detailed description of how both, liveness and readiness probes works.

    If you want to update your deployment with zero downtime you should check Rolling updates.

    Additional resources: