azure-devops cloud-foundry azure-spring-boot azure-spring-cloud

azure-spring-boot-starter-keyvault-secrets not resolving vcap service variables during application startup in PCF

I am upgrading my application from older version of azure-key-vault to latest version of azure bom.

Below are the changes in pom.xml and the app fails to start in PCF

Before making changes in pom.xml:

    <dependency>
      <groupId>com.microsoft.azure</groupId>
      <artifactId>azure-keyvault</artifactId>
      <version>1.2.4</version>
    </dependency>
    <dependency>
      <groupId>com.microsoft.azure</groupId>
      <artifactId>azure-keyvault-secrets-spring-boot-starter</artifactId>
      <scope>runtime</scope>
      <version>2.1.3</version>
    </dependency>
    <dependency>
      <groupId>com.microsoft.azure</groupId>
      <artifactId>azure-storage</artifactId>
      <version>5.5.0</version>
    </dependency>

After making changes in pom.xml


    <dependency>
      <groupId>com.azure.spring</groupId>
      <artifactId>azure-spring-boot-starter-keyvault-secrets</artifactId>
    </dependency>
    .....
    <dependencyManagement>
         <dependency>
            <groupId>com.azure.spring</groupId>
            <artifactId>azure-spring-boot-bom</artifactId>
            <version>3.5.0</version>
            <type>pom</type>
            <scope>import</scope>
          </dependency>
        </dependencies>
      </dependencyManagement>

When using older version, the app starts and was able to read the

azure.keyvault.enabled=true
azure.keyvault.client-id=${vcap.services.myservice.credentials.clientId}
azure.keyvault.client-key=${vcap.services.myservice.credentials.clientKey}
azure.keyvault.uri=${vcap.services.myservice.credentials.uri}
azure.keyvault.tenant-id=1234567-1234-1234-1234-1234werewrew

property on application startup

But the same is not getting resolved after making the pom.xml changes and getting the below exception

2021-07-22T00:15:38.945-07:00 [CELL/0] [OUT] Downloaded droplet (167.3M)
2021-07-22T00:15:42.169-07:00 [APP/PROC/WEB/0] [OUT] 2021-07-22 07:15:42.166 INFO 15 --- [ main] pertySourceApplicationContextInitializer : 'cloud' property source added
2021-07-22T00:15:42.170-07:00 [APP/PROC/WEB/0] [OUT] 2021-07-22 07:15:42.170 INFO 15 --- [ main] nfigurationApplicationContextInitializer : Reconfiguration enabled
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] 2021-07-22 07:15:51.846 ERROR 15 --- [ main] o.s.boot.SpringApplication : Application run failed
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] java.lang.IllegalStateException: Failed to configure KeyVault property source
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at com.azure.spring.keyvault.KeyVaultEnvironmentPostProcessorHelper.addKeyVaultPropertySource(KeyVaultEnvironmentPostProcessorHelper.java:121) ~[azure-spring-boot-3.5.0.jar:na]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at com.azure.spring.keyvault.KeyVaultEnvironmentPostProcessor.postProcessEnvironment(KeyVaultEnvironmentPostProcessor.java:47) ~[azure-spring-boot-3.5.0.jar:na]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.env.EnvironmentPostProcessorApplicationListener.onApplicationEnvironmentPreparedEvent(EnvironmentPostProcessorApplicationListener.java:100) ~[spring-boot-2.4.8.jar:2.4.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.env.EnvironmentPostProcessorApplicationListener.onApplicationEvent(EnvironmentPostProcessorApplicationListener.java:86) ~[spring-boot-2.4.8.jar:2.4.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:176) ~[spring-context-5.3.8.jar:5.3.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:169) ~[spring-context-5.3.8.jar:5.3.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:143) ~[spring-context-5.3.8.jar:5.3.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:131) ~[spring-context-5.3.8.jar:5.3.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.context.event.EventPublishingRunListener.environmentPrepared(EventPublishingRunListener.java:82) ~[spring-boot-2.4.8.jar:2.4.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.SpringApplicationRunListeners.lambda$environmentPrepared$2(SpringApplicationRunListeners.java:63) ~[spring-boot-2.4.8.jar:2.4.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at java.util.ArrayList.forEach(ArrayList.java:1259) ~[na:1.8.0_272]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:117) ~[spring-boot-2.4.8.jar:2.4.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:111) ~[spring-boot-2.4.8.jar:2.4.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.SpringApplicationRunListeners.environmentPrepared(SpringApplicationRunListeners.java:62) ~[spring-boot-2.4.8.jar:2.4.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.SpringApplication.prepareEnvironment(SpringApplication.java:375) [spring-boot-2.4.8.jar:2.4.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.SpringApplication.run(SpringApplication.java:333) [spring-boot-2.4.8.jar:2.4.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.SpringApplication.run(SpringApplication.java:1329) [spring-boot-2.4.8.jar:2.4.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.SpringApplication.run(SpringApplication.java:1318) [spring-boot-2.4.8.jar:2.4.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at com.TestApplication.main(TestApplication.java:23) [classes/:na]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_272]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_272]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_272]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_272]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) [app/:na]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.loader.Launcher.launch(Launcher.java:108) [app/:na]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.loader.Launcher.launch(Launcher.java:58) [app/:na]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88) [app/:na]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] Caused by: java.lang.RuntimeException: Max retries 3 times exceeded. Error Details: ManagedIdentityCredential authentication unavailable. Connection to IMDS endpoint cannot be established, connect timed out.
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at com.azure.core.http.policy.RetryPolicy.lambda$attemptAsync$1(RetryPolicy.java:127) ~[azure-core-1.16.0.jar:na]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:94) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.MonoFlatMap$FlatMapMain.onError(MonoFlatMap.java:172) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.MonoFlatMap$FlatMapMain.onError(MonoFlatMap.java:172) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.MonoFlatMap$FlatMapMain.secondError(MonoFlatMap.java:192) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.MonoFlatMap$FlatMapInner.onError(MonoFlatMap.java:259) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onError(MonoPeekTerminal.java:258) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onError(MonoPeekTerminal.java:258) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxPeekFuseable$PeekFuseableConditionalSubscriber.onError(FluxPeekFuseable.java:553) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxPeekFuseable$PeekFuseableConditionalSubscriber.onError(FluxPeekFuseable.java:553) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onError(MonoPeekTerminal.java:258) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxPeekFuseable$PeekFuseableConditionalSubscriber.onError(FluxPeekFuseable.java:553) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onError(MonoPeekTerminal.java:258) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.Operators.error(Operators.java:197) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxFlatMap.trySubscribeScalarMap(FluxFlatMap.java:135) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.MonoFlatMap.subscribeOrReturn(MonoFlatMap.java:53) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:57) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:157) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:120) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:120) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:199) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.SerializedSubscriber.onNext(SerializedSubscriber.java:99) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.onNext(FluxRetryWhen.java:173) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.Operators$MonoInnerProducerBase.complete(Operators.java:2663) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.MonoSingle$SingleSubscriber.onComplete(MonoSingle.java:180) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.MonoFlatMapMany$FlatMapManyInner.onComplete(MonoFlatMapMany.java:260) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onComplete(FluxMapFuseable.java:150) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onComplete(FluxMapFuseable.java:150) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.MonoCollect$CollectSubscriber.onComplete(MonoCollect.java:159) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.onComplete(FluxDoFinally.java:145) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxHandle$HandleSubscriber.onComplete(FluxHandle.java:212) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxMap$MapConditionalSubscriber.onComplete(FluxMap.java:269) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.netty.channel.FluxReceive.onInboundComplete(FluxReceive.java:401) ~[reactor-netty-core-1.0.8.jar:1.0.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.netty.channel.ChannelOperations.onInboundComplete(ChannelOperations.java:420) ~[reactor-netty-core-1.0.8.jar:1.0.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.netty.channel.ChannelOperations.terminate(ChannelOperations.java:474) ~[reactor-netty-core-1.0.8.jar:1.0.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:685) ~[reactor-netty-http-1.0.8.jar:1.0.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:94) ~[reactor-netty-core-1.0.8.jar:1.0.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.65.Final.jar:4.1.65.Final]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.65.Final.jar:4.1.65.Final]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.65.Final.jar:4.1.65.Final]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) ~[netty-transport-4.1.65.Final.jar:4.1.65.Final]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) ~[netty-codec-4.1.65.Final.jar:4.1.65.Final]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) ~[netty-codec-4.1.65.Final.jar:4.1.65.Final]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) ~[netty-transport-4.1.65.Final.jar:4.1.65.Final]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.65.Final.jar:4.1.65.Final]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.65.Final.jar:4.1.65.Final]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.65.Final.jar:4.1.65.Final]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1368) ~[netty-handler-4.1.65.Final.jar:4.1.65.Final]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1245) ~[netty-handler-4.1.65.Final.jar:4.1.65.Final]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1282) ~[netty-handler-4.1.65.Final.jar:4.1.65.Final]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507) ~[netty-codec-4.1.65.Final.jar:4.1.65.Final]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446) ~[netty-codec-4.1.65.Final.jar:4.1.65.Final]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.65.Final.jar:4.1.65.Final]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.65.Final.jar:4.1.65.Final]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.65.Final.jar:4.1.65.Final]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.65.Final.jar:4.1.65.Final]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.65.Final.jar:4.1.65.Final]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.65.Final.jar:4.1.65.Final]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.65.Final.jar:4.1.65.Final]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.65.Final.jar:4.1.65.Final]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795) ~[netty-transport-native-epoll-4.1.65.Final-linux-x86_64.jar:4.1.65.Final]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480) ~[netty-transport-native-epoll-4.1.65.Final-linux-x86_64.jar:4.1.65.Final]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) ~[netty-transport-native-epoll-4.1.65.Final-linux-x86_64.jar:4.1.65.Final]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[netty-common-4.1.65.Final.jar:4.1.65.Final]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.65.Final.jar:4.1.65.Final]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.65.Final.jar:4.1.65.Final]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_272]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] Suppressed: java.lang.Exception: #block terminated with an error
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:99) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.Flux.blockLast(Flux.java:2519) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at com.azure.core.util.paging.ContinuablePagedByIteratorBase.requestPage(ContinuablePagedByIteratorBase.java:94) ~[azure-core-1.16.0.jar:na]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at com.azure.core.util.paging.ContinuablePagedByPageIterable$ContinuablePagedByPageIterator.<init>(ContinuablePagedByPageIterable.java:49) ~[azure-core-1.16.0.jar:na]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at com.azure.core.util.paging.ContinuablePagedByPageIterable.iterator(ContinuablePagedByPageIterable.java:37) ~[azure-core-1.16.0.jar:na]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at java.lang.Iterable.spliterator(Iterable.java:101) ~[na:1.8.0_272]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at com.azure.spring.keyvault.KeyVaultOperation.lambda$refreshProperties$2(KeyVaultOperation.java:142) ~[azure-spring-boot-3.5.0.jar:na]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at java.util.Optional.map(Optional.java:215) ~[na:1.8.0_272]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at com.azure.spring.keyvault.KeyVaultOperation.refreshProperties(KeyVaultOperation.java:142) ~[azure-spring-boot-3.5.0.jar:na]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at com.azure.spring.keyvault.KeyVaultOperation.<init>(KeyVaultOperation.java:80) ~[azure-spring-boot-3.5.0.jar:na]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at com.azure.spring.keyvault.KeyVaultEnvironmentPostProcessorHelper.addKeyVaultPropertySource(KeyVaultEnvironmentPostProcessorHelper.java:101) ~[azure-spring-boot-3.5.0.jar:na]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at com.azure.spring.keyvault.KeyVaultEnvironmentPostProcessor.postProcessEnvironment(KeyVaultEnvironmentPostProcessor.java:47) ~[azure-spring-boot-3.5.0.jar:na]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.env.EnvironmentPostProcessorApplicationListener.onApplicationEnvironmentPreparedEvent(EnvironmentPostProcessorApplicationListener.java:100) ~[spring-boot-2.4.8.jar:2.4.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.env.EnvironmentPostProcessorApplicationListener.onApplicationEvent(EnvironmentPostProcessorApplicationListener.java:86) ~[spring-boot-2.4.8.jar:2.4.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:176) ~[spring-context-5.3.8.jar:5.3.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:169) ~[spring-context-5.3.8.jar:5.3.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:143) ~[spring-context-5.3.8.jar:5.3.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:131) ~[spring-context-5.3.8.jar:5.3.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.context.event.EventPublishingRunListener.environmentPrepared(EventPublishingRunListener.java:82) ~[spring-boot-2.4.8.jar:2.4.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.SpringApplicationRunListeners.lambda$environmentPrepared$2(SpringApplicationRunListeners.java:63) ~[spring-boot-2.4.8.jar:2.4.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at java.util.ArrayList.forEach(ArrayList.java:1259) ~[na:1.8.0_272]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:117) ~[spring-boot-2.4.8.jar:2.4.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:111) ~[spring-boot-2.4.8.jar:2.4.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.SpringApplicationRunListeners.environmentPrepared(SpringApplicationRunListeners.java:62) ~[spring-boot-2.4.8.jar:2.4.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.SpringApplication.prepareEnvironment(SpringApplication.java:375) [spring-boot-2.4.8.jar:2.4.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.SpringApplication.run(SpringApplication.java:333) [spring-boot-2.4.8.jar:2.4.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.SpringApplication.run(SpringApplication.java:1329) [spring-boot-2.4.8.jar:2.4.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.SpringApplication.run(SpringApplication.java:1318) [spring-boot-2.4.8.jar:2.4.8]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at com.TestApplication.main(TestApplication.java:23) [classes/:na]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_272]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_272]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_272]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_272]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) [app/:na]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.loader.Launcher.launch(Launcher.java:108) [app/:na]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.loader.Launcher.launch(Launcher.java:58) [app/:na]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88) [app/:na]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] Caused by: com.azure.identity.CredentialUnavailableException: ManagedIdentityCredential authentication unavailable. Connection to IMDS endpoint cannot be established, connect timed out.
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at com.azure.identity.implementation.IdentityClient.lambda$checkIMDSAvailable$50(IdentityClient.java:1208) ~[azure-identity-1.3.0.jar:na]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.MonoCallable.call(MonoCallable.java:91) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] at reactor.core.publisher.FluxFlatMap.trySubscribeScalarMap(FluxFlatMap.java:127) ~[reactor-core-3.4.7.jar:3.4.7]
2021-07-22T00:15:51.847-07:00 [APP/PROC/WEB/0] [OUT] ... 55 common frames omitted
2021-07-22T00:15:51.848-07:00 [APP/PROC/WEB/0] [OUT] Caused by: java.net.SocketTimeoutException: connect timed out
2021-07-22T00:15:51.848-07:00 [APP/PROC/WEB/0] [OUT] at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_272]
2021-07-22T00:15:51.848-07:00 [APP/PROC/WEB/0] [OUT] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_272]
2021-07-22T00:15:51.848-07:00 [APP/PROC/WEB/0] [OUT] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_272]
....
2021-07-22T00:15:51.848-07:00 [APP/PROC/WEB/0] [OUT] at com.azure.identity.implementation.IdentityClient.lambda$checkIMDSAvailable$50(IdentityClient.java:1202) ~[azure-identity-1.3.0.jar:na]
2021-07-22T00:15:52.202-07:00 [APP/PROC/WEB/0] [OUT] Exit status 1
2021-07-22T00:15:52.204-07:00 [CELL/SSHD/0] [OUT] Exit status 0

Has anything changed which is breaking this?

Solution

Finally I was able to resolve the issue

Few properties are defined with camel case as below clientId and clientKey in my application.properties file and those are causing the issue.

azure.keyvault.client-id=${vcap.services.myservice.credentials.clientId}
azure.keyvault.client-key=${vcap.services.myservice.credentials.clientKey}

After changing the place holders from camel case to lower case, the app was able to resolve those properties. I changed the place holders as below

azure.keyvault.client-id=${vcap.services.myservice.credentials.clientid}
azure.keyvault.client-key=${vcap.services.myservice.credentials.clientkey}

NOTE: I did not change to lower case in PCF service bindings and they are still with camel case. So, seems like spring-boot is converting all the keys to lower case