[SOLVED] Memory leaks while authenticating user with oauth2 and keycloak (SSO)

We have recently migrated app from Vaadin 14 to Vaadin 24 (Spring boot, Vaadin-flow, java 17).

We are using keycloak+oauth2 login with SSO as we did before. This works very simple:

user opens URL www.app.com
we generate JWK based on our certificate
we redirect user to keycloak, then based on user roles and keycloak mappers user is either logged in or not, no login/password required

This all worked fine in previous version of app but now we experience a memory leak while trying to authorize 1st user who is trying to open the URL after app starts.

It looks like browser is continuously trying to establish SSL session before we create the JWK authotize user which is causing this error:

class=o.a.t.u.n.j.JSSESupport level=DEBUG thread=https-jsse-nio-8443-exec-8 trace= login=- - Error trying to obtain a certificate from the client
    javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
        at java.base/sun.security.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:1034)
        at org.apache.tomcat.util.net.jsse.JSSESupport.getPeerCertificateChain(JSSESupport.java:107)
        at org.apache.coyote.AbstractProcessor.populateSslRequestAttributes(AbstractProcessor.java:815)
        at org.apache.coyote.AbstractProcessor.action(AbstractProcessor.java:489)
        at org.apache.coyote.Request.action(Request.java:523)
        at org.apache.catalina.connector.Request.getAttribute(Request.java:851)
        at org.apache.catalina.connector.Request.getAttributeNames(Request.java:925)
        at org.apache.catalina.connector.RequestFacade.getAttributeNames(RequestFacade.java:257)
        at org.springframework.web.servlet.handler.HandlerMappingIntrospector$AttributesPreservingRequest.initAttributes(HandlerMappingIntrospector.java:489)
        at org.springframework.web.servlet.handler.HandlerMappingIntrospector$AttributesPreservingRequest.<init>(HandlerMappingIntrospector.java:483)
        at org.springframework.web.servlet.handler.HandlerMappingIntrospector.setCache(HandlerMappingIntrospector.java:216)
        at org.springframework.web.servlet.handler.HandlerMappingIntrospector.lambda$createCacheFilter$3(HandlerMappingIntrospector.java:193)
        at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:113)
        at org.springframework.web.filter.CompositeFilter.doFilter(CompositeFilter.java:74)
        at org.springframework.security.config.annotation.web.configuration.WebMvcSecurityConfiguration$CompositeFilterChainProxy.doFilter(WebMvcSecurityConfiguration.java:230)
class=o.a.t.u.n.N.handshake level=DEBUG thread=https-jsse-nio-8443-exec-6 trace=3de145a2ae27eb0c login=- - Handshake failed for client connection from IP address [0:0:0:0:0:0:0:1] and port [59728]
java.io.IOException: An established connection was aborted by the software in your host machine
    at java.base/sun.nio.ch.SocketDispatcher.read0(Native Method)
    at java.base/sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:46)
    at java.base/sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:330)
    at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:296)
    at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:259)
    at java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:417)
    at org.apache.tomcat.util.net.SecureNioChannel.handshakeUnwrap(SecureNioChannel.java:467)
    at org.apache.tomcat.util.net.SecureNioChannel.handshake(SecureNioChannel.java:214)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1716)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1190)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63)
    at java.base/java.lang.Thread.run(Thread.java:842)
2024/07/08 14:49:53,547 class=o.a.t.u.n.NioEndpoint level=ERROR thread=https-jsse-nio-8443-exec-6 trace=3de145a2ae27eb0c login=- - Failed to close channel
java.io.IOException: Invalid close state, will not send network data.
    at org.apache.tomcat.util.net.SecureNioChannel.close(SecureNioChannel.java:544)
    at org.apache.tomcat.util.net.SecureNioChannel.close(SecureNioChannel.java:559)
    at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doClose(NioEndpoint.java:1212)
    at org.apache.tomcat.util.net.SocketWrapperBase.close(SocketWrapperBase.java:425)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1748)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1190)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63)
    at java.base/java.lang.Thread.run(Thread.java:842)

This error appears 3-4 times and results in 3-4 byte arrays stuck in memory. Each one taking 1gb of heap. We have profiled the app with JProfiler and this is the result of incoming references of one of the byte arrays:

I have tried to debug the SSL session establishment but without luck.

This is the relevant security configuration of the Vaadin 24 version:

@Configuration
@EnableWebSecurity
public class KeycloakSecurityConfig extends VaadinWebSecurity {

    private OAuth2AccessTokenResponseClient<OAuth2AuthorizationCodeGrantRequest> getAuthorizationCodeToken() {
        DefaultAuthorizationCodeTokenResponseClient authorizationCodeTokenResponseClient = new DefaultAuthorizationCodeTokenResponseClient();
        OAuth2AuthorizationCodeGrantRequestEntityConverter oAuth2AuthorizationCodeGrantRequestEntityConverter = new OAuth2AuthorizationCodeGrantRequestEntityConverter();
        oAuth2AuthorizationCodeGrantRequestEntityConverter.addParametersConverter(new NimbusJwtClientAuthenticationParametersConverter((cc) -> {
            try {
                Key key = keyStore.getKey(alias, keyPassword);
                X509Certificate xcert = (X509Certificate) temp.getCertificate(keyId);
                RSAKey rsaJWK = RSAKey.parse(xcert);
                JWK jwk = new RSAKey.Builder(rsaJWK)
                        .keyID(keyId)
                        .privateKey((RSAPrivateKey) key)
                        .keyUse(KeyUse.SIGNATURE)
                        .keyStore(temp)
                        .build();
                return jwk;
            } catch (Exception e) {
                log.error(e);
            }
            return null;
        }));
        authorizationCodeTokenResponseClient.setRequestEntityConverter(oAuth2AuthorizationCodeGrantRequestEntityConverter);
        return authorizationCodeTokenResponseClient;
    }

    @Override
    protected void configure(HttpSecurity http) throws Exception {
        http.authorizeHttpRequests(req -> req.requestMatchers(AntPathRequestMatcher.antMatcher("/status")).permitAll());
        super.configure(http);
        http.headers(header -> header.addHeaderWriter(new XFrameOptionsHeaderWriter(XFrameOptionsHeaderWriter.XFrameOptionsMode.SAMEORIGIN)));
        http
                .httpBasic(AbstractHttpConfigurer::disable)
                .formLogin(AbstractHttpConfigurer::disable)
                .anonymous(AbstractHttpConfigurer::disable)
                .csrf(AbstractHttpConfigurer::disable)
                .oauth2Login(login -> login
                        .tokenEndpoint(endpoint -> endpoint.accessTokenResponseClient(getAuthorizationCodeToken()))
                        .userInfoEndpoint(userInfoEndpointConfig -> userInfoEndpointConfig
                                .userAuthoritiesMapper(userAuthoritiesMapper())
                                .oidcUserService(userRequest -> {
                                    return mapUser(userRequest);
                                })));
    }
}

The leak is not getting larger on subsequent users even though the errors also appear. What can be wrong here with the configuration?

EDIT: Sorry, I've missed some logs on initial post, thanks for pointing it out. I have added the IOException to the logs - that is probably causing the memory leak. On subsequent user logins only the 1st error about the handshake appears. There is no "IOException" anymore. Garbage collector is not collecting the byte arrays, they just stay referenced in the memory as shown in the Jprofiler screenshot. We were getting OutOfMemoryError but we increased heap to debug it.

When the app starts heap size is about 200mb, then after first user logs in it goes up to 4.4gb which is definietly not a normal behaviour.