spring-bootspring-boot-actuatorjava-21

Integrate k8s secret live reload with spring-boot 3 and spring-cloud-k8s-config-watcher


I am trying to integrate the live reloading of k8s secrets in my spring boot 3 application.

My setup is as follows:

I have a Java 21 application that is being build as a Docker image containing an executable jar. The docker image is deployed on a k8s cluster using Helm and a secret is mounted in my deployment.yaml as follows:

        - name: LIVE_RELOAD_SECRET
          valueFrom:
            secretKeyRef:
              name: {{ .Chart.Name }}-live-secrets
              key: TEST_LIVERELOAD_SECRET

The secret is present when I deploy (if I exec into the pod and do echo $LIVE_RELOAD_SECRET I get the expected value).

So far so good.

Now I want to be able to read this secret in my application AND also change it at runtime. I tried multiple solutions:

    @RestController
    @RefreshScope
    public class SecretLogger {
    
        private final PropertyResolver propertyResolver;
        private String liveReloadSecret;
        private String springCloudPropery;
    
        @Autowired
        public SecretLogger(@Value("${live-reload-secret}") String liveReloadSecret, @Value("${some.other.property}") String springCloudPropery, PropertyResolver propertyResolver) {
            this.liveReloadSecret = liveReloadSecret;
            this.propertyResolver = propertyResolver;
            this.springCloudPropery = springCloudPropery;
        }
    
        @GetMapping("/secret")
        public String liveReload() {
    
            return "Injected value: " + liveReloadSecret + "\n" +
                    "PropertyResolver: " + propertyResolver.getProperty("live-reload-secret") + "\n" +
                    "System.getEnv: " + System.getenv("LIVE_RELOAD_SECRET") + "\n" +
                    "Spring cloud property: " + springCloudPropery;
    
        }
    }

I want to be able to call /refresh on my actuator endpoint and see the values change (by calling the /secret endpoint afterwards). I also added some.other.property, a spring-cloud property as a check (we also use spring-cloud-config-server for env specific properties).

The spring-cloud-config property:

some.other.property=someTestValue

So when everything is up and running and I call the /secret endpoint I first get this output:

    Injected value: OYALELE
    PropertyResolver: OYALELE
    System.getEnv: OYALELE
    Spring cloud property: someTestValue

This is in line with what I expect. The k8s secret has the value OYALELE and the spring-cloud-config value is also someTestValue.

Next I change the value of the secret in k8s and also the value in spring-cloud-config and I do a post to the /refresh endpoint on my deployed application.

The result is that the spring-cloud-config property has changed value, but the k8s secret seems to be the same. The only explanation I can think of is that you can not "live reload" env vars in a java application. But then I don't know how this https://docs.spring.io/spring-cloud-kubernetes/docs/current/reference/html/#monitoring-configmaps-and-secrets should ever work for secrets?

And I am not even using the configwatcher yet (that would be the next step). I don't see any reason to try the config watcher, as all it would do is automate the calling of the /refresh endpoint for me on changes to the secret. But if it does not work when I do it manually it will probably also not work otherwise.

I think I am just missing some logic or building blocks to live reload secrets on a k8s deployed spring boot 3 app but I can't get my head around them...

Any help would be much appreciated!


Solution

  • With the help of this thread I managed to get all of it working.

    I will explain my mistakes and link an example project that uses kubernetes secrets and configmaps as properties and is able to refresh them after calling the /actuator/refresh endpoint.

    My first mistake was trying to use "env" values in my deployment in k8s. Spring has no way of "updating" these with a context refresh as far as I know. What I ended up using was mounted secrets. Meaning that you mount the secret as a file on your container. To get this working you need to use 'spring.config.import: "configtree:/mnt/secrets/"', telling spring to look under a specified folder for properties. Alternatively or in combination you could also use 'spring.config.import: "configtree:/mnt/secrets/,kubernetes:,optional:configserver:"' if you want to use multiple property sources.

    The "kubernetes:" part is used if you want to use the kubernetes discoveryClient to get secrets and configMaps, but you need the right RBAC rights for this (see example project).

    Once I got this right, everything else started working as well. My example project does not include spring-cloud-kubernetes-config-watcher but once the /actuator/refresh endpoint works, spring-cloud-kubernetes-config-watcher should be a straight forward implementation following the docs.

    My full implementation also included a SecretProviderClass that synced secrets from AzureKeyVault so I opted for file based properties because the SecretProviderClass already provides this implementation.