I'm using an autopilot cluster and have some workloads that can't be stopped under 10 minutes. I would like to set a termination grace period to X hours/days to prevent any interruption.
When I set manually a terminationGracePeriodSeconds over 600 seconds, it's automatically set back to 600 with a warning: Warning: autopilot-default-resources-mutator:The max supported TerminationGracePeriodSeconds is 600 seconds. Defaulting down from configured 1800 seconds to 600 seconds. deployment.apps/extended-pods configured
In Google documentation I saw this annotation to use to prevent this behavior cluster-autoscaler.kubernetes.io/safe-to-evict: "false" introduced in July, 2023 and documented here
But, when I'm using this annotation, I sill have the warning and the update of the TerminationGracePeriodSeconds.
Am I missing something?
As per this official document
In GKE For Autopilot clusters therterminationGracePeriodSeconds is limited to 10 minutes (600 seconds) for most Pods except for Spot Pods, which are limited to 25 seconds.
The Annotation which you applied prevents the cluster Autoscaler from evicting a pod during Scale down. But it does not override the termination grace period enforced by Autopilot.
As per this kodekloud blog You can also use a preStop hook in your pods lifecycle to cleanup tasks before termination.
Still you require a longer grace period to consider requesting extended run time of pods through google cloud support to inquire about specific feature requests for your use case.