java spring spring-boot spring-cloud spring-cloud-config

How to manage multiple .properties files in Spring Boot for different deploying environments?

I have a Spring Boot application where in the src/main/resources directory I have multiple .properties files which I'm using something like this given below via code:

@PropertySource(value = "classpath:properties/myfunctional.properties”)

Currently, I'm able to achieve my work with this as it's a simple functionality.

But the thing is keeping all these .properties files in src/main/resources is not a good idea because of the following reasons:

These .properties files will be part of the build and packaged into a .jar file.
I guess storing sensitive information like apikey, token in the codebase via properties is not a good practice.

So, I have few questions around this:

How and where can I move all of my .properties files to an external server?
How to roll out property changes during the deployment as these values are going to differ based on the environment like dev, test, stage & prod?

Note: Few of the techniques I have tried with Spring Boot Config, Git Secret Variable and so on. But again I'm not sure on how much secure these techniques are.

So, I'm looking for an answer which will cover broader aspects.

Solution

You can achieve your requirements via Spring Cloud Config Server and Client.

Please refer to this doc here.

Github Approach:

Step-By-Step Process where a Github repo will be acting as a source for all external properties files containing confidential/sensitive data.

I have created and hosted all my secrets-{env}.properties files on my Github repo for testing and demonstration.

secrets-dev.properties:

secrets.api-key=<dev-api-key>
secrets.token=<dev-token>

secrets-stage.properties:

secrets.api-key=<stage-api-key>
secrets.token=<stage-token>

secrets-test.properties:

secrets.api-key=<test-api-key>
secrets.token=<test-token>

secrets-prod.properties:

secrets.api-key=<prod-api-key>
secrets.token=<prod-token>

You have to create and setup the Spring Boot Cloud Config Server (acting as an external server) which will take the responsibility of connecting to the Github external repository and provide the necessary way for the clients to get access to the properties files based on the environment.

Project Structure:

pom.xml:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>3.1.4</version>
        <relativePath /> <!-- lookup parent from repository -->
    </parent>
    <groupId>com.example</groupId>
    <artifactId>sb-externalize-config-server</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>sb-externalize-config-server</name>
    <description>Demo project for Spring Boot</description>
    <properties>
        <java.version>17</java.version>
        <spring-cloud.version>2022.0.4</spring-cloud.version>
    </properties>
    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-config-server</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-config</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-configuration-processor</artifactId>
            <optional>true</optional>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
    </dependencies>
    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.springframework.cloud</groupId>
                <artifactId>spring-cloud-dependencies</artifactId>
                <version>${spring-cloud.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>

</project>

application.properties:

spring.cloud.config.server.git.uri=https://github.com/anishb266/secrets-repo
server.port=8081

Key points to be noted:

You have to set spring.cloud.config.server.git.uri=https://github.com/anishb266/secrets-repo to connect to that repo.

Startup Class:

package com.example;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.config.server.EnableConfigServer;

@SpringBootApplication
@EnableConfigServer
public class SbExternalizeConfigServerApplication {

    public static void main(String[] args) {
        SpringApplication.run(SbExternalizeConfigServerApplication.class, args);
    }

}

You have to use @EnableConfigServer to make it an external config server.

Now, I will be running the external config server on port 8081.

Output:

Accessing one of the properties, i.e, secrets-test.properties like this:

http://localhost:8081/{application}/{env} 

application - secrets
env - test

Config Server Log:

You have to create and setup the Spring boot Cloud Config client which will connect to config server and get the properties value based on the environment.

Project Structure:

pom.xml:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>3.1.4</version>
        <relativePath/> <!-- lookup parent from repository -->
    </parent>
    <groupId>com.example</groupId>
    <artifactId>sb-externalize-config-client</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>sb-externalize-config-client</name>
    <description>Demo project for Spring Boot</description>
    <properties>
        <java.version>17</java.version>
        <spring-cloud.version>2022.0.4</spring-cloud.version>
    </properties>
    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-config</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-configuration-processor</artifactId>
            <optional>true</optional>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
    </dependencies>
    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.springframework.cloud</groupId>
                <artifactId>spring-cloud-dependencies</artifactId>
                <version>${spring-cloud.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>

</project>

application.properties:

spring.config.import=optional:configserver:http://localhost:8081
server.port=8080
spring.application.name=secrets
spring.profiles.active=test

Key points to be noted here:

You have set the spring.application.name with the application name (here it's secrets) before -{env}.properties for looking up into config server. Otherwise, it won't work. Basically, it tries to find properties file via name of the application. By default, the name is application only. So, it will try to find application-{env}.properties by default.
For testing, I'm keeping spring.profiles.active=test so that client will try to fetch the properties from secrets-test.properties via config server.
You have set this spring.config.import=optional:configserver:http://localhost:8081 for connecting to the external server.

SecretsProperties:

package com.example;

import org.springframework.boot.context.properties.ConfigurationProperties;

@ConfigurationProperties("secrets")
public class SecretsProperties {

    private String apiKey;

    private String token;

    public SecretsProperties(String apiKey, String token) {
        this.apiKey = apiKey;
        this.token = token;
    }

    @Override
    public String toString() {
        return "SecretsProperties [apiKey=" + apiKey + ", token=" + token + "]";
    }

    public String getApiKey() {
        return apiKey;
    }

    public void setApiKey(String apiKey) {
        this.apiKey = apiKey;
    }

    public String getToken() {
        return token;
    }

    public void setToken(String token) {
        this.token = token;
    }

}

This class is to map the loaded properties.

Startup class:

package com.example;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.context.properties.EnableConfigurationProperties;

@SpringBootApplication
@EnableConfigurationProperties(SecretsProperties.class)
public class SbExternalizeConfigClientApplication {

    public static void main(String[] args) {
        SpringApplication.run(SbExternalizeConfigClientApplication.class, args);
    }

}

TestController for testing:

package com.example;

import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
public class TestController {

    private SecretsProperties properties;

    public TestController(SecretsProperties properties) {
        this.properties = properties;
    }

    @GetMapping("test-external-properties")
    public SecretsProperties testExternalProperties() {
        return properties;
    }

}

Now, I'm running the client at 8080 with profiles test.

Client Log:

Output:

Providing a part of the External Config server log to proof that client was able to pick the desired properties with test profile via config server with the application-name secrets:

2023-10-16T18:04:48.433+05:30  INFO 79167 --- [nio-8081-exec-2] o.s.c.c.s.e.NativeEnvironmentRepository  : Adding property source: Config resource 'file [/var/folders/yq/m5hjv94j18586b1gzwzjg1tw0000gn/T/config-repo-5291131953887905898/secrets-test.properties]' via location 'file:/var/folders/yq/m5hjv94j18586b1gzwzjg1tw0000gn/T/config-repo-5291131953887905898/'

That's all.

HashiCorp Vault Approach:

If you are looking for a more secured and encrypted way of saving properties, then go for HashiCorp Vault.

Note: Vault acts as an external server for keeping secrets. So, you don't need to create an external server from scratch.

Read the Spring Boot docs here on how to do it.

From Docs:

Vault is a secrets management system allowing you to store sensitive data which is encrypted at rest. It’s ideal to store sensitive configuration details such as passwords, encryption keys, API keys.

Process is quite similar but there will some changes.

First download Vault for here.
Extract it. There will be only one file vault.sh.
Follow the installation process here.
For demonstation purposes, I have used this command -> ./vault server --dev --dev-root-token-id="abc" where I have taken an example token as abc. Please don't run like this in prod.

Open another console and execute this command.

  export VAULT_TOKEN="abc"
  export VAULT_ADDR="http://127.0.0.1:8200"

Enter the values to vault. Format:

  ./vault kv put secret/{application-name}/{env} key=value

Example commands to put data into vault:

  ./vault kv put secret/secrets/test secrets.api-key=something secrets.token=something

  ./vault kv put secret/secrets/dev secrets.api-key=nothing secrets.token=nothing

I will be using the same existing client project to demonstrate.

Add this dependency in the pom.xml:

<dependency>
     <groupId>org.springframework.cloud</groupId>
     <artifactId>spring-cloud-starter-vault-config</artifactId>. 
</dependency>

You don't need this dependency below anymore:

    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-config</artifactId>
    </dependency>

Kindly remove this.

pom.xml will look like this:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>3.1.4</version>
        <relativePath/> <!-- lookup parent from repository -->
    </parent>
    <groupId>com.example</groupId>
    <artifactId>sb-externalize-config-client</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>sb-externalize-config-client</name>
    <description>Demo project for Spring Boot</description>
    <properties>
        <java.version>17</java.version>
        <spring-cloud.version>2022.0.4</spring-cloud.version>
    </properties>
    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-vault-config</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-configuration-processor</artifactId>
            <optional>true</optional>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
    </dependencies>
    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.springframework.cloud</groupId>
                <artifactId>spring-cloud-dependencies</artifactId>
                <version>${spring-cloud.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>

</project>

Set application.properties with the below properties:

spring.application.name=secrets
spring.cloud.vault.token=abc
spring.cloud.vault.scheme=http
spring.cloud.vault.kv.enabled=true
spring.cloud.vault.kv.default-context=
spring.config.import=vault://secret/${spring.application.name}/${spring.profiles.active}
spring.profiles.active=test

Points to be noted:

By default, the spring cloud starter vault picks and appends the /application as application name to lookup for the secrets from vault. To prevent this, we need to add spring.cloud.vault.kv.default-context= property.

The TestController, SecretsProperties and SbExternalizeConfigClientApplication remain the same.

Output:

I'm running the client for test profile.

Note: This was a very basic approach that I showed with Vault.

Read docs for more info on how to do more advanced setup for Vault.