I have a grails 6 application, built using gradlew war and deployed on Tomcat 9 (AWS). The application was converted from Grails 2.5.6, with a fresh build. With the change to grails 6, I converted from using <g:remoteFunction> on the gsps to using ajax calls. Locally, and for 2-3 hrs in production after the application server is started, the ajax calls run successfully, and the app behaves as expected. However, after this time, unless the app server is restarted, I get a rash of these errors below, and the users then struggle with disappearing elements that would be updated by the ajax return results.
My suspicion is that the spring security session expires, and something strange is going on when the authenticationfilter(s), or other filters, maybe exception/error handling fire. If there is an exception within the ajax call (controller or service exception) I would not expect the ErrorPageFilter to be called (as I want to handle the exception, and return a 503 or other "bad" response to Ajax, complete with an error that is usefult to the user (or to me).
I have tried: Increasing the spring session timeout to 8 hours in aaplication.yaml: spring: session: timeout: 28800
Removing logging of the login time against the User record from grails.plugin.springsecurity.onInteractiveAuthenticationSuccessEvent
Catching ALL exceptions in the controller method(s) called by Ajax (the only one caught is the "ControllerExecutionException"):
@Transactional
def theAjaxMethod() {
log.error "ajax params ${params} for theAjaxMethod ${getPrincipal()?.username}"
if (check_logged_in()) {
try {
// do stuff including update the db,
// some methods call a service,
//some do direct domainObject.save flush:true failOnError:true
// then render the template piece for the return to Ajax.
render template: '/folder/template', model: [// model stuff for template here//]
} catch (Exception ex) {
if (ex instanceof ControllerExecutionException) {
log.error "ajax Caught a ControllerExecutionException in method (the method) logged in user is ${getPrincipal()?.username}"
} else {
def errorForDisplay = "ajax addInvoiceDetail : error: ${ex}, params: ${params} "
log.error "****** ajaxController errros!! ${errorForDisplay}"
render(status: 503, text: errorForDisplay)
}
}
} else {
log.error "Not logged in when calling add invoice Detail"
render(status: 503, text: notLoggedInError)
}
}
Any help with debugging this further would be greatly appreciated. I'm sure there is some exception happening underneath, but untill I can see what it is, I have no way to be able to handle it and fix it.
Regards Brigette
Here is the full stacktrace of a recent error: Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: 2024-11-04 14:25:37.196 ERROR --- [io-8080-exec-12] StackTrace : Full Stack Trace: Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: java.lang.IllegalStateException: The response object has been recycled and is no longer associated with this facade Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.apache.catalina.connector.ResponseFacade.checkFacade(ResponseFacade.java:504) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.apache.catalina.connector.ResponseFacade.isCommitted(ResponseFacade.java:270) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.apache.catalina.connector.ResponseFacade.setContentType(ResponseFacade.java:213) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at javax.servlet.ServletResponseWrapper.setContentType(ServletResponseWrapper.java:134) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at javax.servlet.ServletResponseWrapper.setContentType(ServletResponseWrapper.java:134) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at javax.servlet.ServletResponseWrapper.setContentType(ServletResponseWrapper.java:134) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at javax.servlet.ServletResponseWrapper.setContentType(ServletResponseWrapper.java:134) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.grails.web.taglib.encoder.WebOutputContextLookup$WebOutputContext.setContentType(WebOutputContextLookup. java:130) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.grails.gsp.GroovyPageWritable.doWriteTo(GroovyPageWritable.java:124) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.grails.gsp.GroovyPageWritable.writeTo(GroovyPageWritable.java:82) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.grails.web.servlet.view.GroovyPageView.renderTemplate(GroovyPageView.java:76) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.grails.web.servlet.view.AbstractGrailsView.renderWithinGrailsWebRequest(AbstractGrailsView.java:71) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.grails.web.servlet.view.AbstractGrailsView.renderMergedOutputModel(AbstractGrailsView.java:55) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.springframework.web.servlet.view.AbstractView.render(AbstractView.java:316) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at grails.artefact.controller.support.ResponseRenderer$Trait$Helper.renderViewForTemplate(ResponseRenderer.groo vy:615) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at grails.artefact.controller.support.ResponseRenderer$Trait$Helper.render(ResponseRenderer.groovy:354) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at grails.artefact.controller.support.ResponseRenderer$Trait$Helper$render$3.call(Unknown Source) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at com.hailsystem.AjaxController.render(AjaxController.groovy) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at grails.artefact.controller.support.ResponseRenderer$render$0.callCurrent(Unknown Source) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at com.hailsystem.AjaxController.$tt__detailChanged(AjaxController.groovy:358) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at jdk.internal.reflect.GeneratedMethodAccessor893.invoke(Unknown Source) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at java.base/java.lang.reflect.Method.invoke(Method.java:566) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1268) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1035) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.codehaus.groovy.runtime.InvokerHelper.invokePogoMethod(InvokerHelper.java:1029) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:1012) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.codehaus.groovy.runtime.InvokerHelper.invokeMethodSafe(InvokerHelper.java:101) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at com.hailsystem.AjaxController$_detailChanged_closure5.doCall(AjaxController.groovy) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at jdk.internal.reflect.GeneratedMethodAccessor892.invoke(Unknown Source) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at java.base/java.lang.reflect.Method.invoke(Method.java:566) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:274) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1035) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at groovy.lang.Closure.call(Closure.java:412) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at groovy.lang.Closure.call(Closure.java:428) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at grails.gorm.transactions.GrailsTransactionTemplate$2.doInTransaction(GrailsTransactionTemplate.groovy:94) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at grails.gorm.transactions.GrailsTransactionTemplate.execute(GrailsTransactionTemplate.groovy:91) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at com.hailsystem.AjaxController.detailChanged(AjaxController.groovy) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.grails.core.DefaultGrailsControllerClass$MethodHandleInvoker.invoke(DefaultGrailsControllerClass.java:22 3) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.grails.core.DefaultGrailsControllerClass.invoke(DefaultGrailsControllerClass.java:188) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.grails.web.mapping.mvc.UrlMappingsInfoHandlerAdapter.handle(UrlMappingsInfoHandlerAdapter.groovy:90) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1072) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:965) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at javax.servlet.http.HttpServlet.service(HttpServlet.java:555) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at javax.servlet.http.HttpServlet.service(HttpServlet.java:623) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:199) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:337) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor .java:115) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityIntercept or.java:81) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java: 122) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java: 116) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at grails.plugin.springsecurity.web.UpdateRequestContextHolderExceptionTranslationFilter.doFilter(UpdateRequest ContextHolderExceptionTranslationFilter.groovy:64) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at grails.plugin.springsecurity.web.filter.GrailsAnonymousAuthenticationFilter.doFilter(GrailsAnonymousAuthenti cationFilter.groovy:54) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberM eAuthenticationFilter.java:106) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberM eAuthenticationFilter.java:97) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContext HolderAwareRequestFilter.java:149) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuth enticationProcessingFilter.java:223) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuth enticationProcessingFilter.java:217) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at grails.plugin.springsecurity.web.authentication.logout.MutableLogoutFilter.doFilter(MutableLogoutFilter.groo vy:64) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenc eFilter.java:112) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenc eFilter.java:82) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at grails.plugin.springsecurity.web.SecurityRequestHolderFilter.doFilter(SecurityRequestHolderFilter.groovy:58) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:221) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:186) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.grails.web.servlet.mvc.GrailsWebRequestFilter.doFilterInternal(GrailsWebRequestFilter.java:77) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.grails.web.filters.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:67) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilte r.java:96) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.springframework.boot.web.servlet.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:126) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.springframework.boot.web.servlet.support.ErrorPageFilter.access$000(ErrorPageFilter.java:64) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.springframework.boot.web.servlet.support.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:101) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.springframework.boot.web.servlet.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:119) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:91) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:168) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:482) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:130) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:761) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:660) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:346) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:388) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:936) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1791) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1190) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63) Nov 4 04:25:37 ip-172-31-24-38 server[2083867]: #011at java.base/java.lang.Thread.run(Thread.java:829)
What exact version of Tomcat is being used? We also saw this with Tomcat 9.0.93 and a Grails 4 application.
In our research we found a couple of potential changelog entries that were relevant:
https://tomcat.apache.org/tomcat-9.0-doc/changelog.html
The RECYCLE_FACADES system property might be able to mitigate this but I haven't tested it. We used Tomcat 9.0.86 and moved on for now.