[SOLVED] Failed to install jupyterhub(z2jh) with helm on k8s, giving hook-image-awaiter with timeout error (Error: pre-upgrade hooks failed)

Failed to install jupyterhub(z2jh) with helm on k8s, giving hook-image-awaiter with timeout error (Error: pre-upgrade hooks failed)

I'm installing jupyterhub on k8s using helm.

helm upgrade --cleanup-on-fail --install jupyterhub jupyterhub-2.0.0/jupyterhub/ --namespace my-NS --create-namespace --version=2.0.0 --values my-values.yaml --timeout 30m --debug

Its failing with error in creating hook-image-awaiter pods.

Error from helm debug:

upgrade.go:142: [debug] preparing upgrade for jupyterhub
upgrade.go:150: [debug] performing update for jupyterhub
upgrade.go:322: [debug] creating upgraded release for jupyterhub
client.go:310: [debug] Starting delete for "hook-image-puller" DaemonSet
client.go:128: [debug] creating 1 resource(s)
client.go:310: [debug] Starting delete for "hook-image-awaiter" ServiceAccount
client.go:128: [debug] creating 1 resource(s)
client.go:310: [debug] Starting delete for "hook-image-awaiter" Job
client.go:128: [debug] creating 1 resource(s)
client.go:540: [debug] Watching for changes to Job hook-image-awaiter with timeout of 30m0s
client.go:568: [debug] Add/Modify event for hook-image-awaiter: ADDED
client.go:607: [debug] hook-image-awaiter: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:568: [debug] Add/Modify event for hook-image-awaiter: MODIFIED
client.go:607: [debug] hook-image-awaiter: Jobs active: 1, jobs failed: 1, jobs succeeded: 0
client.go:568: [debug] Add/Modify event for hook-image-awaiter: MODIFIED
client.go:607: [debug] hook-image-awaiter: Jobs active: 1, jobs failed: 2, jobs succeeded: 0
client.go:568: [debug] Add/Modify event for hook-image-awaiter: MODIFIED
client.go:607: [debug] hook-image-awaiter: Jobs active: 1, jobs failed: 3, jobs succeeded: 0
client.go:568: [debug] Add/Modify event for hook-image-awaiter: MODIFIED
client.go:607: [debug] hook-image-awaiter: Jobs active: 1, jobs failed: 4, jobs succeeded: 0
client.go:568: [debug] Add/Modify event for hook-image-awaiter: MODIFIED
client.go:607: [debug] hook-image-awaiter: Jobs active: 1, jobs failed: 5, jobs succeeded: 0
client.go:568: [debug] Add/Modify event for hook-image-awaiter: MODIFIED
client.go:607: [debug] hook-image-awaiter: Jobs active: 1, jobs failed: 6, jobs succeeded: 0
client.go:568: [debug] Add/Modify event for hook-image-awaiter: MODIFIED
upgrade.go:434: [debug] warning: Upgrade "jupyterhub" failed: pre-upgrade hooks failed: job failed: BackoffLimitExceeded
Error: UPGRADE FAILED: pre-upgrade hooks failed: job failed: BackoffLimitExceeded
helm.go:84: [debug] pre-upgrade hooks failed: job failed: BackoffLimitExceeded
UPGRADE FAILED

There are some answers in web which didn't help to resolve the issue. I have tried following,

Increase helm install time out - didn't work
Failed to pull image: I can see jupyterhub/k8s-image-awaiter:2.0.0 got pulled and I do not see any errors when pods are described

hook-image-puller: hook-image-awaiter:

Event logs:

PS: I disabled rbac in values.yaml (create=false) and am installing jupyterhub in a new namespace while there is one is running in another namespace)

Solution