[SOLVED] How to add truststore and keystore to a springboot app in PCF(Pivotal Cloud Foundry)

As you mentioned, there are a number of ways you can do this and it depends on your specific needs as to which one will work best. I'll try to outline as many options as possible and pros/cons of each.

If all you need are to distribute certificates to trust, then adding Bosh Trusted certificates and allowing Bosh to distribute them to all of the VMs & Containers in CF is a good way to go. In Ops Manager, under the Bosh tile, go to Security and you can add in any number of trusted certificates.

This is a good approach when multiple apps need to trust the certificates, like with a corporate/private CA. It also works well with different environments/foundations, because you have a different Ops Manager which can maintain a different list of trusted certs. It can be difficult to manage if you have many certificates that need trusted and/or if certificates are only used by one or a small number of apps. It also requires administrative privileges to CF & can take a while to deploy updates across an entire foundation.
This is the option you mentioned in your question. Import the certificate to a Java truststore file, pack the file into the Java application and specify its path via the JAVA_OPTS environment variable. The truststore file can be placed under the resource directory.

Ex: cf set-env <app> JAVA_OPTS '-Djavax.net.ssl.trustStore=/home/vcap/app/BOOT-INF/classes/jssecacerts'

This option is useful for scenarios not covered by option #1 above, like single or small applications, where there are certificates unique to that application. For example, you have a database server you must trust and only one app uses that database server.

It has the downfall that you mentioned, the truststore must be embedded into the application, so to handle multiple environments you'd need to do something like package all of the environment's certificates into one truststore and package that into a single JAR/WAR for all environments or build multiple JAR/WAR files, one for each environment.

It also has the downfall that you're completely overriding the default truststore. The default truststore comes populated with a lot of well-known CA certs, so you must start with a copy of the default truststore and append the certificates you wish to add. Otherwise, you will lose the default list of trusted CA's, and validation will break to sites like Google or Yahoo. The default CA cert comes with every JRE, but it can vary from version to version so you should keep this up-to-date.
This is a slight variation on #2. Create a script file under the application root directory named .profile. In it, run keytool and import all the certs that are packaged with your application.

Ex:
```
#!/bin/bash 
$HOME/.java-buildpack/open_jdk_jre/bin/keytool \
    -keystore $HOME/.java-buildpack/open_jdk_jre/lib/security/cacerts \
    -storepass changeit \
    -importcert \
    -noprompt \
    -alias MyCert \
    -file $HOME/BOOT-INF/classes/ssl/MyCert.crt
# TODO: repeat this command for all the certs you need to import
```
Then run jar uf target/your-app.jar .profile to add the script to the root of your application JAR/WAR. The script must exist in the root of your JAR/WAR file to be picked up by Cloud Foundry and executed. You can run jar tf target/your-app.jar | grep profile to confirm this. The output should indicate .profile with no leading path. If there is a leading path, the file was added in the wrong place.

The benefit of this approach is that you don't need to provide a full trust store. That is a big downside of #2, and in this case, you will neatly use the default truststore packaged with the JRE provided by the Java buildpack. The script will append the additional certs prior to your application starting up.

The downside of this approach is that adding the certificate is messy and can easily be skipped/forgotten causing your app to break. It also needs to be in a very specific place, so that Cloud Foundry will find it and execute it. The downside of needing to package your certs into the JAR/WAR is also present with this option.
The next option is an iteration on #3. In this case, you include the .profile script but do not package the certs into your JAR/WAR. Instead, you supply the certificates via an environment variable or bound service (this works nicely with the CredHub service broker).

To make this work, you'll need to adjust the script so that it takes the certificate out of the environment variable, makes a temporary file, and imports that into the default truststore with keytool. The following example show taking it out of an environment variable, but you could use jq and pull it out of VCAP_SERVICES for a bound service.

Ex:
```
#!/bin/bash 
$HOME/.java-buildpack/open_jdk_jre/bin/keytool \
    -keystore $HOME/.java-buildpack/open_jdk_jre/lib/security/cacerts \
    -storepass changeit \
    -importcert \
    -noprompt \
    -alias MyCert \
    -file <(echo $CERT_1)
# TODO: repeat this command for all the certs you need to import
```
The benefits of this approach are the same as with #3, except that you no longer need to bundle certificates with your application. They can be fed externally. The downside is the same, the .profile script is tricky to work with and can be forgotten.
A variation on #2 is to set the system properties in your code. You just need this to happen early in the start-up of the application before the JVM initializes TLS. If it happens after, nothing will happen and your certs won't be found.

This article explains the various places you can put initialization code in a Spring Boot app. Basically, you're just using that to call System.setProperty("javax.net.ssl.trustStore", "path/to/custom/truststore") and if you change the password, System.setProperty("javax.net.ssl.trustStorePassword", "newpassword").

I'm not a fan of this approach because things can unexpectedly break if TLS is initialized for before your code runs. IMHO, it's tricky to control this, especially in Spring Boot with all its auto configurations (not that these are bad, just that you have to be really careful about order). I find it just easier to do this in the profile script which is guaranteed to run before your app starts.
Another variation would be to configure your WebClient manually and not depend on the default truststore at all. You can create your own truststore, which gives you the ultimate flexibility.

Ex:
```
@Bean
public WebClient createWebClient() throws SSLException {
    SslContext sslContext = SslContextBuilder
            .forClient()
            .trustManager( /* TODO */ )
            .build();
    ClientHttpConnector httpConnector = HttpClient.create().secure(t -> t.sslContext(sslContext) )
    return WebClient.builder().clientConnector(httpConnector).build();
}
```
There is a TODO marker in the code above, where you need to add your trust manager. This SO post has some good thoughts on how to do that, but overall what you create and where you load certs from is quite flexible.

The upshot of this method is that it's totally customizable. You can pull in your certificates from anywhere, environment variables, VCAP_SERVICES, or even from Spring Cloud Config. You just need them handy when you initialize your WebClient. The downside is probably obvious, it's more code to write.

There are probably other options but hopefully, this will get you thinking about what is possible. The above information also only touches on the subject of truststores. If you need to provide a Keystore with private keys, some of the options above will work with slight differences.

Won't work.
It Will work, but has a different JVM argument: javax.net.ssl.keyStore. 3 and 4. Can work, but instead of appending to a default keystore, you must create a new one in the script and also set javax.net.ssl.keyStore. It's more work and not a lot more helpful than #2.
It will work with the same caveats, but has a different JVM argument: javax.net.ssl.keyStore.
It will work, you just need to code it.

You also need to be more careful with keys because they are highly sensitive information. IMHO, it would be worth the extra effort to make sure they are not stored in a JAR/WAR and kept secure like in CredHub or Spring Cloud Config Server.