We're trying to secure a connection from a Java-program to a remote server. The server is presenting a certificate signed by our own CA, and our Java-program is rejecting that:
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
at sun.security.ssl.Alerts.getSSLException(Alerts.java:154)
at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:2023)
The error only happens with older JDK8 patch-levels, however: 8u111. With 8u401 everything works...
The $JAVAHOME/jre/lib/security/cacerts is a symlink to the same file for both older and newer JDK-versions, and that file does contain our CAs' certificates, according to keytool -list:
Certificate fingerprint (SHA-256): 57:4D:F6:93:1E:27:80:39:66:7B.....
OURCOMPANY
Certificate fingerprint (SHA-256): 83:81:CF:C1:68:99:29:F0:D2:C1.....
OURCOMPANY
Is there anything else, something about the certificates themselves, that makes them unusable with the older JDK? Is it the digest-method (SHA-256), that the older JDK does not recognize -- I see, that the cacerts file, that came with the older version originally, used 1024-bit RSA for fingerprinting...
Is there a means to somehow enable the usage of SHA-256 without upgrading the entire JDK (which we cannot do at the moment for unrelated reasons)?
Can we, for example, import our CAs' certificates into the older KeyStore -- using the RSA 1024 fingerprinting, or is the digest-method determined by the certificate itself?
You probably need to download and activate the Unlimited Strength Jurisdiction Policy Files from Oracle.
Effectively you can turn on and off the stronger encryption in Java with them. (You will need an Oracle Account for the download.)
The linked page says
The unlimited policy files are required only for JDK 8, 7, and 6 updates earlier than 8u161, 7u171, and 6u181. On those versions and later, the stronger cryptographic algorithms are available by default.
This would fit with your observation that 8u111 does not perform the handshake while 8u401 does.