javasslsslengine

Using SSLEngine to use a certificate signed for an internal network location


Background

I'm trying to create 2 java applications that talk to each other over SSL on an internal network. I'm doing this by giving an SSLEngine access to a keystore that contains a private key entry for the certificate to be used.

   private static SSLEngine createSslEngine()
   {
      try
      {
         KeyStore ks = KeyStore.getInstance( KeyStore.getDefaultType() );
         char[] pwdArray = "changeit".toCharArray();
         ks.load( null, pwdArray );
         try ( InputStream ksIs = ServerAssociationConnector.class.getResourceAsStream( "clientkeystore" ) )
         {
            ks.load( ksIs, pwdArray  );
         }
         KeyManagerFactory kmf = KeyManagerFactory.getInstance( KeyManagerFactory.getDefaultAlgorithm() );

         kmf.init( ks, pwdArray );
         SSLContext context = SSLContext.getInstance( "TLSv1.3" );
         context.init( kmf.getKeyManagers(), null, null );

         return context.createSSLEngine();
      }
      catch ( NoSuchAlgorithmException | UnrecoverableKeyException | KeyStoreException | CertificateException | IOException | KeyManagementException exception )
      {
         throw new AssertionError( "SSLException", exception );
      }
   }

This works fine when using root certificates (hitting dubrovnik:2762 in a browser complains about the untrusted certificate but otherwise works). However; I would like to use a non root certificate that is signed by an internal CA just for the domain in question (called dubrovnik in this example).

When I instead sign a certificate with the a root certificate and have dubrovnik in the subject alternative name java refuses to use the certificate and instead produces a javax.net.ssl.SSLHandshakeException: No available authentication scheme exception

enter image description here

The keystore I'm using contains this:

Keystore type: PKCS12
Keystore provider: SUN

Your keystore contains 2 entries

dubrovnik, 2 Nov 2020, PrivateKeyEntry,
Certificate fingerprint (SHA-256): 88:5F:53:7C:85:8E:65:01:3E:E1:E8:F0:D6:17:7A:8B:22:EF:11:DD:5F:E6:30:FE:A7:3B:F1:FA:07:C8:46:38
thecaroot, 2 Nov 2020, trustedCertEntry,
Certificate fingerprint (SHA-256): 9E:2F:86:B6:17:83:D2:26:88:42:49:E4:3F:DA:DA:19:31:11:18:F7:15:6D:16:35:C4:3E:1B:E4:F8:E6:FC:3A

How I've been creating my certificates

I've been using openssl and keytool to create the local ca and signed certificate

openssl genrsa -des3 -out myCA.key 2048
openssl req -x509 -new -nodes -key myCA.key -sha256 -days 1825 -out myCA.pem

##create the certificate to sign
keytool -keystore clientkeystore -genkey -alias dubrovnik -dname "CN=dubrovnik" -ext san=dns:dubrovnik
##create a request to have certificate signed
keytool –certreq –keystore clientkeystore –alias dubrovnik –keyalg rsa –file dubrovnik.csr

##CREATE A FILE CALLED dubrovnik.conf as described in https://stackoverflow.com/a/47779814/2187042

##sign the request
openssl x509 -req -CA myCA.pem -CAkey myCA.key -in dubrovnik.csr -out dubrovnik.cer -days 3650 -CAcreateserial -extfile dubrovnik.conf -extensions v3_req
##import the root level certificate (just the certificate, not the private key)
keytool -import -keystore clientkeystore -file myCA.pem -alias theCARoot 
##import the signed client certificate
keytool -import -keystore clientkeystore -file dubrovnik.cer -alias dubrovnik

Question

What do I need to do to make an internal network certificate (not root certificate) acceptable to SSLEngine and avoid a javax.net.ssl.SSLHandshakeException: No available authentication scheme exception


Solution

  • I believe my problem was actually just that the root certificate was an RSA certificate but the non-root certificate was DSA, and DSA is deprecated (and so was being ignored).

    This problem was also encountered at javax.net.ssl.SSLHandshakeException: No available authentication scheme

    The DSA nature of the certificate can be seen in the certificate details

    enter image description here

    The keygen command needs to be altered to use RSA

    keytool -keystore clientkeystore -genkey -alias dubrovnik –keyalg rsa