JWT Validation Failure in Spring Boot API due to PKIX Path Issue with Keycloak on GKE (Gateway API, TLS Termination)

I have a GKE architecture with the following components:

1. External Traffic Flow:

   • Public F5 load balancer (terminates TLS for *.example.com) routes to:
     • app.example.com, api.example.com, idp.example.com
   • Internal GKE load balancer (uses wildcard cert *.k8s.internal.example) forwards to a Gateway (Gateway API).
   • Gateway routes HTTP traffic to respective services via HTTPRoute resources (hostnames include both public and internal domains).

2. Keycloak (IDP) Setup:

   • Keycloak deployed in the cluster (via Operator) with a ClusterIP service (idp-custom-service:8443).
   • Configured with a realm and required SSL certificates.

3. Spring Boot API (simeox-api):

   • Uses Spring Security OAuth2 Resource Server for JWT validation.
   • Configured with issuer-uri: https://idp.example.com/realms/my-realm.

Error Observed:
The API fails to validate JWT tokens, throwing:

org.springframework.security.oauth2.jwt.JwtDecoderInitializationException:  
Caused by: SSLHandshakeException: PKIX path building failed: unable to find valid certification path  

The root cause is the API cannot retrieve Keycloak's OIDC configuration from https://idp.example.com/realms/my-realm/.well-known/openid-configuration due to certificate trust issues.

Investigation:

• The API pod’s trust store does NOT trust the certificate presented by idp.example.com.
• The internal LB’s certificate (*.k8s.internal.example) is likely not trusted by the API’s JVM, but the public F5 certificate (*.example.com) is publicly trusted.
• Keycloak’s service uses a self-signed or internal certificate (mounted via Kubernetes secrets).

Questions:

1. How should TLS be configured end-to-end to ensure the API pod trusts Keycloak’s certificate when the internal LB terminates TLS with a different certificate?

   • Should the Gateway or Keycloak’s service expose the public F5 certificate instead of the internal LB’s cert?
   • Is there a need to inject the internal LB’s CA certificate into the API pods’ trust store?

2. Keycloak Certificate Setup:

   • Why does Keycloak require a private certificate if TLS is terminated at the F5/LB?
   • Should Keycloak’s HTTPS listener use a self-signed cert, or should it align with the LB’s certificate chain?

Relevant Configuration Snippets:

# HTTPRoute for IDP  
spec:  
  hostnames:  
    - idp.example.com  
  rules:  
    - backendRefs:  
        - name: idp-custom-service  
          port: 8443  

# Keycloak Service  
spec:  
  ports:  
    - name: https  
      port: 8443  
      targetPort: 8443  

Additional Context:

• The API uses Spring Boot 3.3.x with spring-boot-starter-oauth2-resource-server.
• The internal LB’s certificate is not publicly signed (e.g., internal CA or self-signed).

---

Secondary Question (Optional):
When deploying Keycloak, why is it necessary to provide a certificate with a private key if TLS is terminated at the load balancer? Shouldn’t the LB handle SSL offloading? Does Keycloak require HTTPS even for internal communication between the LB and pods?

---

To resolve the issue, I added the Gandi certificate of the F5 load balancer to the backend API if the certificate is publicly signed by Gandi. Initially, I thought it should have been the gateway certificate or the Keycloak certificate, but it turned out that the F5 certificate was actually the one needed.