I am currently struggling connecting to a Kafka cluster the tls-way.
My consumer is a spring-kafka java application (using org.springframework.kafka.core.DefaultKafkaConsumerFactory). Its (related) configuration is below.
I have no problem producing / consuming in a non-TLS way, but encountered a (so classical) "unable to find valid certification path to requested target" otherwise.
The error looks like this :
2024-03-26 18:34:12.292 [kafkaMessageListenerContainer-C-1] ERROR o.s.k.l.KafkaMessageListenerContainer - Authentication/Authorization Exception and no authExceptionRetryInterval set
org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to re
quested target
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
...
at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1175)
at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396)
...
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path
to requested target
at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439)
at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306)
at java.base/sun.security.validator.Validator.validate(Validator.java:264)
...
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:146)
at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:127)
at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434)
... 36 common frames omitted
I tried to validate my truststore directly, bypassing the application ... with success :
$> openssl s_client -verify 100 -showcerts -connect server1.fqdn:9093 -CAfile <(keytool -list -rfc -keystore /some/path/to/my/truststore.p12 -storepass somepwd)
SSL handshake has read 2440 bytes and written 138 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 3072 bit
Secure Renegotiation IS supported
...
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
...
Timeout : 300 (sec)
Verify return code: 0 (ok)
Which makes me think the truststore is correct. My truststore only contains the root certificate. It is emitted by the very same authority that did sign the server certificate. The server keystore only contains its own certificate (with CN=server1.fqdn)
The TLS connexion is 2-ways. As far as I understand (I am not fluent in TLS), the first negociation is made with the client (me) asking the server for its certificate. The latest will then be compared with whatever lies in my truststore (at least authority will be checked with my truststore's root ca). Thus, since my error shows the content of the server's certificate (javax.net.debug=all), I guess that my problem is related to this first negocation, and thus the way the truststore is used.
I tried to provide the truststore as javax.net.ssl.trustStore=/some/path/to/my/truststore.p12 (as it was proposed in several related posts), but I had the same error.
I also tried to add the server certificate within (with its own alias) my trusstore, but while the openssl check does work, the java/kafka consumer does not.
Eventually, I just changed the password (ssl.truststore.password**) of the truststore ( in order to see whether it is used in anyway, and I encounter an other -legitimate- error. According to me, it shows it is (properly?) loaded.
Thus, I am stuck here with what sounds like a classical error (and the cause will surely be a classical and obvious mistake of me)... but I definetely don't know how to work it out!
Kafka : 3.60
Controller : Kraft
Consumer properties:
bootstrap.servers=server1.fqdn:9093,server2.fqdn:9093,server3.fqdn:9093
group.id=stg
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = [hidden]
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = /some/path/to/my/keystore.p12
ssl.keystore.password = [hidden]
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = /some/path/to/my/truststore.p12
ssl.truststore.password = [hidden]
ssl.truststore.type = JKS
security.protocol = SSL
security.providers = null
sasl.mechanism = GSSAPI
As expected, my problem was obvious... or at least it is now that I know it. I was focused on the root certificate, and since the openssl returned "OK", I took for granted that it was.
Actually, looking (with a teammate) to the remote server certificate, we realized the issuer was "server" type and not "root" type. From there, the solution was obvious : importing the server certificate (on top or instead of the root one) into the truststore... and the magic operates!
One side note : the openssl command I thought helped me show whether my truststore was ok or not... did not work for that purpose!