I want to achieve a use-case where, during graceful scale-down, I want to ensure that cadence workers do not accept any new jobs. I am using cadence on k8, so I plan to give a terminationGracePeriodSeconds to a known maximum timeout by which I know that all the in-progress tasks will be finished on that particular pod. Hence new tasks will be allocated only to active workers.
My use case is that my activity has large startToClose timeout and during deployment, the activity task will be picked up and cannot complete until the timeout and retry.
This is the background/use case:
the activity will still wait for start to close timeout and then retry, in scenarios where we have large start to close timeout, how do retry immediatly
It's recommended to set correct timeout/retry policy for activity to mitigate this issue.
For large startToClose timeout, activity should set heartbeat timeout and send heartbeat to server.
Without heartbeat timeout, the issue in the question could happen even this "graceful shutdown" is provided. Because there could be some crash in activity execution, or worker host crashes.