Jenkins agent templates lost after jenkins controller pod restarts

Setting up custom build agents using Kubernetes plugin on Jenkins represents a major issue.

Every time the Jenkins master node restarts, the custom agent templates defined in Manage Jenkins > Configure System > Cloud > Kubernetes are lost and we need to re-add them again.

I wonder what should be done to permanently save our custom agent templates especially that we regularly update Jenkins version and/or plugins.

Is there any solution for this ?

What we tried:

• Add custom agent templates under Manage Jenkins > Configure System > Cloud > Kubernetes
• Perform an action that requires Jenkins to restart.

What is expected

• Custom Jenkins agent templates still there.

What we found

• Jenkins custom agents are lost and need to be manually added again

I got it working by adding my agent templates in podTemplates map in values like the following:

podTemplates:
customAgent: |
  - name: customAgent
    label: customAgent
    imagePullSecrets:
    - name: regcred
    containers:
      - name: agent
        image: agent_docker_image:0.1
        command: sleep
        args: 99999
        ttyEnabled: true
        workingDir: /home/jenkins/agent