java spring-boot spring-boot-actuator spring-boot-3

Actuator endpoints returning 500 error after upgrade to spring boot 3

I am upgrading a service from spring boot 2.7 to 3.2.0 and bumping into an issue where I get an HTTP Status 500 – Internal Server Error error when accessing actuator endpoints.

Looking at the stack trace I can see that when accessing an actuator endpoint via port 9090 the org.springframework.security.config.annotation.web.AbstractRequestMatcherRegistry.DispatcherServletDelegatingRequestMatcher#matcher method fails to find the dispatcherServletRegistration in the this.servletContext.getServletRegistration(name); call.

When accessing normal app endpoints on port 8080, I can see that it finds it just fine.

This is my actuator configuration, besides the defaults update it is unchanged from my spring-boot 2.7 config.

management:
  defaults:
    metrics:
      export:
        enabled: true
  endpoints:
    web:
      exposure:
        include: '*'
  info:
    build:
      enabled: true
  endpoint:
    health:
      show-details: when_authorized
  server:
    port: 9090

During start-up logs seem to indicate that things are setup ok

2023-12-23T21:36:28.729 [main] [] INFO  o.a.coyote.http11.Http11NioProtocol.log - Starting ProtocolHandler ["http-nio-8080"]
2023-12-23T21:36:28.755 [main] [] INFO  o.s.b.w.e.tomcat.TomcatWebServer.start - Tomcat started on port 8080 (http) with context path ''
HOTSWAP AGENT: 21:36:28.756 INFO (org.hotswap.agent.plugin.spring.SpringPlugin) - Spring plugin initialized - Spring core version '6.1.1'
2023-12-23T21:36:28.813 [main] [] INFO  o.s.b.w.e.tomcat.TomcatWebServer.initialize - Tomcat initialized with port 9090 (http)
2023-12-23T21:36:28.813 [main] [] INFO  o.a.coyote.http11.Http11NioProtocol.log - Initializing ProtocolHandler ["http-nio-9090"]
2023-12-23T21:36:28.813 [main] [] INFO  o.a.catalina.core.StandardService.log - Starting service [Tomcat]
2023-12-23T21:36:28.813 [main] [] INFO  o.a.catalina.core.StandardEngine.log - Starting Servlet engine: [Apache Tomcat/10.1.16]

(No difference if starting it up without the hotswap agent)

Quite a lot of other spring-boot 2->3 migration done, perhaps I missed something (perhaps spring-security related)?

Any advice on what the issue might be or how to troubleshoot it greatly appreciated.

Edit1: this is the stack trace when accessing e.g. localhost:9090/actuator

java.lang.IllegalArgumentException: Failed to find servlet [dispatcherServletRegistration] in the servlet context
    org.springframework.util.Assert.notNull(Assert.java:172)
    org.springframework.security.config.annotation.web.AbstractRequestMatcherRegistry$DispatcherServletDelegatingRequestMatcher.matcher(AbstractRequestMatcherRegistry.java:529)
    org.springframework.security.web.access.intercept.RequestMatcherDelegatingAuthorizationManager.check(RequestMatcherDelegatingAuthorizationManager.java:79)
    org.springframework.security.web.access.intercept.RequestMatcherDelegatingAuthorizationManager.check(RequestMatcherDelegatingAuthorizationManager.java:48)
    org.springframework.security.authorization.ObservationAuthorizationManager.check(ObservationAuthorizationManager.java:63)
    org.springframework.security.web.access.intercept.AuthorizationFilter.doFilter(AuthorizationFilter.java:95)
    org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
    org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
    org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
    org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:126)
    org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:120)
    org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
    org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
    org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
    org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:100)
    org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
    org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
    org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
    org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:145)
    org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:101)
    org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
    org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
    org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
    org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:179)
    org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
    org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
    org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
    org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
    org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
    org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
    org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
    org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:227)
    org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:221)
    org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
    org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
    org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
    org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:107)
    org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:93)
    org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
    org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
    org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
    org.springframework.security.web.csrf.CsrfFilter.doFilterInternal(CsrfFilter.java:117)
    org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
    org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
    org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
    org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
    org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:91)
    org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
    org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
    org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
    org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
    org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90)
    org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75)
    org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
    org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
    org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
    org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
    org.springframework.security.web.context.SecurityContextHolderFilter.doFilter(SecurityContextHolderFilter.java:82)
    org.springframework.security.web.context.SecurityContextHolderFilter.doFilter(SecurityContextHolderFilter.java:69)
    org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
    org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
    org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
    org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:62)
    org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
    org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
    org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
    org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
    org.springframework.security.web.session.DisableEncodeUrlFilter.doFilterInternal(DisableEncodeUrlFilter.java:42)
    org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
    org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
    org.springframework.security.web.ObservationFilterChainDecorator$AroundFilterObservation$SimpleAroundFilterObservation.lambda$wrap$0(ObservationFilterChainDecorator.java:323)
    org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:224)
    org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
    org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:233)
    org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:191)
    org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:352)
    org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:268)

Edit2:

Created a brand new Spring Initilizer project and see the following differences which look relevant, but not yet sure why.

Spring Initilizer new project:

2023-12-26T18:23:07.912Z  INFO 43176 --- [2)-192.168.0.31] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring DispatcherServlet 'dispatcherServlet'
2023-12-26T18:23:07.912Z  INFO 43176 --- [2)-192.168.0.31] o.s.web.servlet.DispatcherServlet        : Initializing Servlet 'dispatcherServlet'
2023-12-26T18:23:07.913Z  INFO 43176 --- [2)-192.168.0.31] o.s.web.servlet.DispatcherServlet        : Completed initialization in 0 ms
2023-12-26T18:23:13.567Z  INFO 43176 --- [nio-9090-exec-1] o.a.c.c.C.[Tomcat-1].[localhost].[/]     : Initializing Spring DispatcherServlet 'dispatcherServletRegistration'
2023-12-26T18:23:13.567Z  INFO 43176 --- [nio-9090-exec-1] o.s.web.servlet.DispatcherServlet        : Initializing Servlet 'dispatcherServletRegistration'
2023-12-26T18:23:13.567Z  INFO 43176 --- [nio-9090-exec-1] o.s.web.servlet.DispatcherServlet        : Completed initialization in 0 ms

My app

2023-12-26T18:24:40.958 [RMI TCP Connection(4)-192.168.0.31] [] INFO  o.a.c.c.C.[Tomcat].[localhost].[/].log - Initializing Spring DispatcherServlet 'dispatcherServlet'
2023-12-26T18:24:40.958 [RMI TCP Connection(4)-192.168.0.31] [] INFO  o.s.web.servlet.DispatcherServlet.initServletBean - Initializing Servlet 'dispatcherServlet'
2023-12-26T18:24:40.960 [RMI TCP Connection(4)-192.168.0.31] [] INFO  o.s.web.servlet.DispatcherServlet.initServletBean - Completed initialization in 1 ms

Cheers, Mike

Solution

Short-short version

For actuator do not use .requestMatcher("/actuator/health") instead use MvcRequestMatcher.Builder#pattern

Slightly longer version

After creating a small poc to troubleshoot; my issue boiled down to changes in how to add request matchers to the SecurityFilterChain.

The old code was simply using something like .requestMatchers("/actuator/health").permitAll() which isn't the correct way anymore.

org.springframework.security.web.access.intercept.RequestMatcherDelegatingAuthorizationManager#check(java.util.function.Supplier, jakarta.servlet.http.HttpServletRequest) Iterates over list of configured request matchers.

When it comes across a requestMatcher("/path") like matcher the underlying request matcher used is of type org.springframework.security.config.annotation.web.AbstractRequestMatcherRegistry.DispatcherServletDelegatingRequestMatcher.

The matcher logic then does the following where it fails (registration is null)

@Override
public MatchResult matcher(HttpServletRequest request) {
    String name = request.getHttpServletMapping().getServletName();
    ServletRegistration registration = this.servletContext.getServletRegistration(name); <-- Doesn't find a registration
    Assert.notNull(registration, "Failed to find servlet [" + name + "] in the servlet context");
    if (isDispatcherServlet(registration)) {
        return this.mvc.matcher(request);
   }
    return this.ant.matcher(request);
}

I solved my problem by defining this bean to create a different Matcher

@Bean
MvcRequestMatcher.Builder mvc(HandlerMappingIntrospector introspector) {
    return new MvcRequestMatcher.Builder(introspector);
}

Which I inject into my SecurityFilterChain bean creation method and use like this

.requestMatchers(mvc.pattern("/actuator/health"), mvc.pattern("/actuator/info")).permitAll()

I've also had all sorts of other issues

For static resources I use the antMatcher, for example antMatcher("/css/**").
Don't think old .requestMatchers(PathRequest.toStaticResources().atCommonLocations()).permitAll() is working anymore.

I'm still wrapping my head around this and after having tried a ridiculous amount of variations, I've not yet fully grokked the different ways of specifying request matchers but hopefully above will be of use to someone else.