Jetty Invalid SNI issue even when certificate is correct

Our product has a server that uses embedded Jetty (version 11.0.11) providing functionality with clients connecting through HTTP calls. It can be configured for HTTPS and a lot of our customers use it with this configuration. Recently we came across an issue from one of our customers where they see a strange issue with the https configuration. The request sometimes works correctly and sometimes throws an Invalid SNI error from Jetty. We are aware that this type of issue usually happens when the SNI sent in the request does not match what is in the certificate. So in this case, as part of the troubleshooting exercises, we collected wireshark captures to look at what is being sent and compare with what is on the certificate.

The certificate itself has the server in the subject alternate name as something like "server.xyz.abc.com". We are seeing from wireshark logs that the TLS layer is sometimes sending the SNI as "server.xyz.abc.com" while some other times it sends it as "SERVER.xyz.abc.com". Notice the server name portion in uppercase.

The request is succeeding when the TLS layer is sending the SNI as "server.xyz.abc.com", and the request seems to fail when the SNI comes in "SERVER.xyz.abc.com".

We get the message "org.eclipse.jetty.http.BadMessageException: 400: Invalid SNI".

To fix this, we then had the customer fix the certificate to add "SERVER.xyz.abc.com" in the subject alternate name as well, and deployed this certificate on the server. So the certificate now contains:

server.xyz.abc.com

SERVER.xyz.abc.com

However this still hasn't fixed the issue and the Invalid SNI message still keeps coming intermittently even though wireshark shows the Client Hello SNI coming in exactly as it is in the certificate. One thing we noticed is that the failure still happens when the uppercase name comes in as the SNI in the Client Hello request.

I have the following questions:

Is there any known issue with Jetty with the case sensitivity of SNI, and is it the right thing to add the uppercase server name to the certificate?
Why does the TLS layer alternate between sending an uppercase and lowercase name, and is there any type of setting somewhere to force it to send it in say, lowercase?
As a worst case, we may need to disable the SNI check to fix this. Is that the only way to do this, as this would require us to supply code changes to disable this check. From looking at other posts, I think there is no configurable parameter to disable this check in Jetty.

Please help with any ideas you may have, as we are not sure how to proceed with helping our customer.

[UPDATE]: On further digging by enabling Jetty debug logging, we found that the SNI host is coming in as null. So this call inside the Jetty code (SecureRequestCustomizer.java) to get the value of the SNI host is getting back a null for "sniHost" and causing the exception. Any ideas why this would be null and what we can do about this?

protected void checkSni(Request request, SSLSession session)
{
    if (isSniRequired() || isSniHostCheck())
    {
        String sniHost = (String)session.getValue(SslContextFactory.Server.SNI_HOST);

        X509 x509 = getX509(session);
        if (x509 == null)
            throw new BadMessageException(400, "Invalid SNI");
        String serverName = Request.getServerName(request);
        if (LOG.isDebugEnabled())
            LOG.debug("Host={}, SNI={}, SNI Certificate={}", serverName, sniHost, x509);

        if (isSniRequired() && (sniHost == null || !x509.matches(sniHost)))
            throw new BadMessageException(400, "Invalid SNI");

        if (isSniHostCheck() && !x509.matches(serverName))
            throw new BadMessageException(400, "Invalid SNI");
    }
}

Solution

We found the issue after a lot of further digging. The issue was the Java version that was being used. The issue was happening with the Oracle Java 11 (+28) version, and when our customer switched to to using Adoptium 11 it went away.

Below are the details of where the issue was:

First, the correct source code from Jetty 11.0.x from where the error originated is below. The exception came up because the certificate was null. For some reason with Oracle java 11, the certificate was coming back as either null or length 0, from the getLocalCertificates() call.

    protected void customize(SSLEngine sslEngine, Request request)
{
    SSLSession sslSession = sslEngine.getSession();

    if (isSniRequired() || isSniHostCheck())
    {
        String sniHost = (String)sslSession.getValue(SslContextFactory.Server.SNI_HOST);
        X509 x509 = (X509)sslSession.getValue(X509_CERT);
        if (x509 == null)
        {
            Certificate[] certificates = sslSession.getLocalCertificates();
            if (certificates == null || certificates.length == 0 || !(certificates[0] instanceof X509Certificate))
                throw new BadMessageException(400, "Invalid SNI");
            x509 = new X509(null, (X509Certificate)certificates[0]);
            sslSession.putValue(X509_CERT, x509);
        }
        String serverName = request.getServerName();
        if (LOG.isDebugEnabled())
            LOG.debug("Host={}, SNI={}, SNI Certificate={}", serverName, sniHost, x509);

        if (isSniRequired() && (sniHost == null || !x509.matches(sniHost)))
            throw new BadMessageException(400, "Invalid SNI");

        if (isSniHostCheck() && !x509.matches(serverName))
            throw new BadMessageException(400, "Invalid SNI");
    }

    request.setAttributes(new SslAttributes(request, sslSession));
}