I'm trying to setup into my kubernetes cluster a spring-boot-admin service (playing with https://github.com/codecentric/spring-boot-admin-runtime-playground). It went quite good: it sees my service, i'm able to see 'Insights' page, 'Logging' page, while other don't work yet.
I want to setup everything to work and right now i'm struggling with 'JVM->Thread dump', and i ran into issues with jackson objectMapper.
The thing is that my services (not the spring-boot-admin one, the others which i want to monitor) use the specific config:
objectMapper.configure(DeserializationFeature.ACCEPT_SINGLE_VALUE_AS_ARRAY, true);
therefore when i request '/actuator/threaddump' in spring-boot-admin, it fails (because spring-boot-admin requests my service, and it returns not valid json) with error below (the error is quite long, so i've shortened it):
2024-11-12 07:37:52.425 DEBUG 1 --- [nio-8080-exec-3] o.apache.coyote.http11.Http11Processor : Error state [CLOSE_NOW] reported while processing request
org.springframework.web.util.NestedServletException: Request processing failed; nested exception is org.springframework.core.codec.DecodingException: JSON decoding error: Cannot deserialize value of type `java.util.ArrayList<java.lang.Object>` from Object value (token `JsonToken.ST
ART_OBJECT`); nested exception is com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize value of type `java.util.ArrayList<java.lang.Object>` from Object value (token `JsonToken.START_OBJECT`)
at [Source: (io.netty.buffer.ByteBufInputStream); line: 1, column: 1]
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1014) ~[spring-webmvc-5.3.30.jar!/:5.3.30]
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898) ~[spring-webmvc-5.3.30.jar!/:5.3.30]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:529) ~[tomcat-embed-core-9.0.82.jar!/:na]
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) ~[spring-webmvc-5.3.30.jar!/:5.3.30]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:623) ~[tomcat-embed-core-9.0.82.jar!/:na]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:209) ~[tomcat-embed-core-9.0.82.jar!/:na]
...
...
Caused by: org.springframework.core.codec.DecodingException: JSON decoding error: Cannot deserialize value of type `java.util.ArrayList<java.lang.Object>` from Object value (token `JsonToken.START_OBJECT`); nested exception is com.fasterxml.jackson.databind.exc.MismatchedInputExcep
tion: Cannot deserialize value of type `java.util.ArrayList<java.lang.Object>` from Object value (token `JsonToken.START_OBJECT`)
at [Source: (io.netty.buffer.ByteBufInputStream); line: 1, column: 1]
at org.springframework.http.codec.json.AbstractJackson2Decoder.processException(AbstractJackson2Decoder.java:242) ~[spring-web-5.3.30.jar!/:5.3.30]
Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Error has been observed at the following site(s):
*__checkpoint <E2><87><A2> Body from GET http://10.0.3.223:8081/actuator/threaddump [DefaultClientResponse]
Original Stack Trace:
at org.springframework.http.codec.json.AbstractJackson2Decoder.processException(AbstractJackson2Decoder.java:242) ~[spring-web-5.3.30.jar!/:5.3.30]
at org.springframework.http.codec.json.AbstractJackson2Decoder.decode(AbstractJackson2Decoder.java:198) ~[spring-web-5.3.30.jar!/:5.3.30]
at org.springframework.http.codec.json.AbstractJackson2Decoder.lambda$decodeToMono$1(AbstractJackson2Decoder.java:179) ~[spring-web-5.3.30.jar!/:5.3.30]
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:125) ~[reactor-core-3.4.33.jar!/:3.4.33]
at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107) ~[reactor-core-3.4.33.jar!/:3.4.33]
...
...
Suppressed: java.lang.Exception: #block terminated with an error
at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:100) ~[reactor-core-3.4.33.jar!/:3.4.33]
at reactor.core.publisher.Mono.block(Mono.java:1742) ~[reactor-core-3.4.33.jar!/:3.4.33]
at de.codecentric.boot.admin.server.web.servlet.InstancesProxyController.instanceProxy(InstancesProxyController.java:125) ~[spring-boot-admin-server-2.7.16.jar!/:2.7.16]
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Unknown Source) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Unknown Source) ~[na:na]
...
...
Caused by: com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize value of type `java.util.ArrayList<java.lang.Object>` from Object value (token `JsonToken.START_OBJECT`)
at [Source: (io.netty.buffer.ByteBufInputStream); line: 1, column: 1]
at com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:59) ~[jackson-databind-2.13.5.jar!/:2.13.5]
at com.fasterxml.jackson.databind.DeserializationContext.reportInputMismatch(DeserializationContext.java:1741) ~[jackson-databind-2.13.5.jar!/:2.13.5]
...
At first glance it seems to be an easy issue: just find the objectMapper and configure ACCEPT_SINGLE_VALUE_AS_ARRAY=true. BUT I JUST CAN'T FIND IT I mean i can't find the one (seems spring-boot-admin have several ones), which responsible for deserializing of requests to other services.
What i tried:
/**
* this one actually is somehow used by spring-boot-admin, since if i configure
* it wrong the whole app stops working properly,
* but doesn't help with my issue
* */
@Bean
public ObjectMapper objectMapper() {
ObjectMapper objectMapper = new ObjectMapper();
objectMapper.enable(DeserializationFeature.ACCEPT_SINGLE_VALUE_AS_ARRAY)
...
/**also tried this*/
@Bean
public Jackson2ObjectMapperBuilder jackson2ObjectMapperBuilder() {
return new Jackson2ObjectMapperBuilder()
.featuresToEnable(DeserializationFeature.ACCEPT_SINGLE_VALUE_AS_ARRAY);
}
/**also tried this*/
@Bean
Jackson2ObjectMapperBuilderCustomizer jackson2ObjectMapperBuilderCustomizer(){
return jacksonObjectMapperBuilder -> {
jacksonObjectMapperBuilder.featuresToEnable(DeserializationFeature.ACCEPT_SINGLE_VALUE_AS_ARRAY);
};
}
/**
* also, from the error stacktrace, i've found that it uses org.springframework.http.codec.json.AbstractJackson2Decoder
* which is actually used by reactive WebFlux library (in my case my spring-boot-admin works in classic: servlet mode, non-reactive)
* therefore i've tried the below, but also without success, it doesn't use it
* */
@Bean
Jackson2JsonDecoder jackson2JsonDecoder(ObjectMapper mapper){
mapper.enable(DeserializationFeature.ACCEPT_SINGLE_VALUE_AS_ARRAY);
return new Jackson2JsonDecoder(mapper)
So, as i understand the issue, it's following: the spring-boot-admin class de.codecentric.boot.admin.server.web.servlet.InstancesProxyController uses reactive stack (Flux,Mono,etc), which uses separated from other app objectMapper, which i can't just find to reconfigure it.
Does anybody know smth about it?
configs i use, pom.xml:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
<version>2.7.17</version>
</dependency>
<!-- Spring Boot Admin -->
<dependency>
<groupId>de.codecentric</groupId>
<artifactId>spring-boot-admin-starter-server</artifactId>
<version>2.7.16</version>
</dependency>
application.yml
server:
port: 8080
max-http-header-size: 65536
forward-headers-strategy: none
spring:
application: # Application-Infos for the Info-Actuator
name: "@pom.artifactId@"
cloud:
kubernetes:
discovery:
# set this to false if running namespaced
all-namespaces: false
# Spring Boot Admin
boot:
admin:
ui:
public-url: https://hello-world.net/spring-boot-admin
# public-url: ${SPRING_BOOT_ADMIN_UI_PUBLIC_URL:http://localhost:8080}
title: ${SPRING_BOOT_ADMIN_UI_TITLE:Spring Boot Admin}
brand: <img src="assets/img/icon-spring-boot-admin.svg"><span>${SPRING_BOOT_ADMIN_UI_TITLE:Spring Boot Admin}</span>
discovery: # Filter discovery to tagged services
instances-metadata:
spring-boot-admin: true # is added as annotation in service.yaml in helm chart
management: # Actuator Configuration
server:
port: 8081
endpoints:
web:
exposure:
include: "*"
endpoint: # Health-Actuator
health:
show-details: always
probes:
enabled: true
add-additional-paths: true
env: # Environment-Actuator
show-values: always # caution: can reveal passwords
configprops: # Configuration-Actuator
show-values: always # caution: can reveal passwords
info: # Info-Actuator
java:
enabled: true
os:
enabled: true
build:
enabled: true
env:
enabled: true
git:
enabled: true
info: # Application-Infos for the Info-Actuator
group: "@pom.groupId@"
artifact: "@pom.artifactId@"
description: "@pom.description@"
version: "@pom.version@"
spring-boot: "@pom.parent.version@"
spring-boot-admin: "@spring-boot-admin.version@"
spring-cloud: "@spring-cloud.version@"
# Tags for the Spring Boot Admin UI
tags:
spring-boot: "@pom.parent.version@"
spring-boot-admin: "@spring-boot-admin.version@"
spring-cloud: "@spring-cloud.version@"
logging: # Logging-File for the Logfile-Actuator
file:
name: "spring-boot-admin.log"
level:
root: DEBUG
org.apache.coyote: TRACE
org.springframework.web: DEBUG
Not sure if this answer will be helpful for anyone, but here are the roots of issue.
The core issue was on the client side, not on spring-boot-admin server side. As @Erik P (thanks a lot for great help!) mentioned in https://github.com/codecentric/spring-boot-admin/issues/3830 it's not possible to reconfigure objectMapper on spring-boot-admin server side.
On the client side there were 2 issues, both related to WebMvcConfigurer. Actuator server creates child WebAppContext, but the issue is that it still 'sees' my WebMvcConfigurer. It applies it to itself, changing the default output. Spring-boot-admin is not able to work with it.
public static class WebMvcConfig implements WebMvcConfigurer {
@Override
public void configureContentNegotiation(ContentNegotiationConfigurer configurer) {
//1. This specific one makes actuator endpoint return 'content-type: application/json' instead of
//'application/vnd.spring-boot.actuator.v2+json;charset=UTF-8' this results spring-boot-admin to process json
//with LegacyEndpointConverter, which then throws that error:
//`java.util.ArrayList<java.lang.Object>` from Object value (token `JsonToken.START_OBJECT`)
configurer.defaultContentType(MediaType.APPLICATION_JSON_UTF8, MediaType.ALL);
...
@Override
public void configureMessageConverters(List<HttpMessageConverter<?>> converters) {
//2. The second issue was related to the fact that we reconfigured default ApplicationContext's ObjectMapper
//(bad for us, don't do like this). Our changes also were changing default behaviour, and spring-boot-admin didn't work.
I was able to solve both this issues by separating application contexts as below:
+---------------------------+ +---------------------------+
| Root WebAppContext | | Root AppContext |
| | +---------------------------+
| contains WebMvcConfigurer | / \
+---------------------------+ PLAN --> / \
| | / \
+---------------------------+ +---------------------------+ +---------------------------+
| Child-actuator | | Child WebAppContext | | Child-actuator |
| WebAppContext | | contains WebMvcConfigurer | | WebAppContext |
| consumes WebMvcConfigurer | +---------------------------+ +---------------------------+
| and breaks spr-boot-admin | PROFIT
+---------------------------+
With above approach the configureContentNegotiation() issue is solved, the configureMessageConverters() issue is more tricky to solve (since ObjectMapper is still located in root ApplicationContext) but also solvable (for actuator i've registered separated WebMvcConfigurer, which was setting up clean, non-changed ObjectMapper) Exact configurations for the above solution see in Separating spring applicationContext: autoconfigure webMvc in child context