[SOLVED] Cannot get mTLS working on a Google Cloud Load Balancer using PrivateCA

I am trying to get mTLS working on a global HTTPS load balancer in Google Cloud, but the client connections are just being rejected with the error "client_cert_validation_failed".

I followed the exact instructions here and here but it appears the trust config supplied just isn't working.

The trust config looks like this:

trustStores:
- trustAnchors:
  - pemCertificate: |
      -----BEGIN CERTIFICATE-----
      MIICWzCCAgKgAwIBAgIUANxN5VlXyQgGZp4uZJi7iXnIivEwCgYIKoZIzj0EAwIw
      NjERMA8GA1UEChMIVmlnaWxhbnQxITAfBgNVBAMTGGxvbmRvbi5hcGlzLnZpcHJv
      Lm9ubGluZTAeFw0yMzA4MjExMDA1NTZaFw0yMzExMTkxMDA1NTVaMDYxETAPBgNV
      BAoTCFZpZ2lsYW50MSEwHwYDVQQDExhsb25kb24uYXBpcy52aXByby5vbmxpbmUw
      WTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAAT/H53qlrvXeBXtrPi8qnigM/YXKKKg
      tfssUk1Vxz+LoVor3DT4duqwyTm94naykqlmV25IzU5BAknff07TGVv5o4HtMIHq
      MA8GA1UdDwEB/wQFAwMH3oAwOwYDVR0lBDQwMgYIKwYBBQUHAwEGCCsGAQUFBwMC
      BggrBgEFBQcDAwYIKwYBBQUHAwQGCCsGAQUFBwMIMBIGA1UdEwEB/wQIMAYBAf8C
      AQUwHQYDVR0OBBYEFHo4y3dPvnRlI622Ytm5x/8WKZ1yMB8GA1UdIwQYMBaAFHo4
      y3dPvnRlI622Ytm5x/8WKZ1yMEYGA1UdEQQ/MD2CH2FnZW50cy5sb25kb24uYXBp
      cy52aXByby5vbmxpbmWCGm1jZC5hZ2VudHMuY2Eudmlwcm8ub25saW5lMAoGCCqG
      SM49BAMCA0cAMEQCIBmn2170sahTzA0iBYuULNeywX+r8fX8JucglsMQNBT8AiAP
      vJwqN9I7bmTgItfpjozMIExZiSeiTn45TfEdNLNsTQ==
      -----END CERTIFICATE-----

NOTE: PEM is definitely correct.

I have a TLS policy which uses this:

gcloud beta network-security server-tls-policies describe london-ips-tls-policy --location=global

createTime: '2023-08-22T15:59:51.030272533Z'
mtlsPolicy:
  clientValidationMode: REJECT_INVALID
  clientValidationTrustConfig: projects/MY_PROJECT_NUM/locations/global/trustConfigs/london-ips-trust-config
name: projects/MY_PROJECT_ID/locations/global/serverTlsPolicies/london-ips-tls-policy
updateTime: '2023-08-22T16:28:04.303764856Z'

I then have a target-https-proxies which references this:

# ...
serverTlsPolicy: //networksecurity.googleapis.com/projects/MY_PROJECT_ID/locations/global/serverTlsPolicies/london-ips-tls-policy

However whenever I download one of the approved certificates like this one:

Using the go code:

// load key/cert pair
    clientCertForTls, err := tls.LoadX509KeyPair(os.ExpandEnv(*clientCertPath), os.ExpandEnv(*privateKeyPath))
    if err != nil {
        log.Fatalf("unable to load x509 pair, %v", err)
    }

    // http client with client certs
    mTLSClient := &http.Client{
        Transport: &http.Transport{
            TLSClientConfig: &tls.Config{
                ClientAuth:   tls.RequestClientCert,
                Certificates: []tls.Certificate{clientCertForTls},
            },
        },
    }

    // make request
    resp, err := mTLSClient.Get("https://OUR_URL.com/")
    if err != nil {
        log.Fatalf("unable to mtls ips, %v", err)
    }
    bodyDat, _ := io.ReadAll(resp.Body)
    log.Printf("body: %s", bodyDat)
    for k := range resp.Header {
        log.Printf("header[%s]: %s", k, resp.Header.Get(k))
    }
    if resp.StatusCode != http.StatusOK {
        log.Fatalf("failed to mtls ips, %s: %s", resp.Status)
    } else {
        log.Printf("ALL GOOD! :)")
    }

However the connection is always rejected by the server.

Using the logging query:

jsonPayload.@type="type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry"

I can see the error "client_cert_validation_failed" in the rejection entry:

{
httpRequest: {
latency: "0s"
remoteIp: "REDACTED"
}
insertId: "REDACTED"
jsonPayload: {
@type: "type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry"
backendTargetProjectNumber: "projects/MY_PROJECT_NUM"
remoteIp: "REDACTED"
statusDetails: "client_cert_validation_failed"
}
logName: "projects/MY_PROJECT_ID/logs/requests"
receiveTimestamp: "2023-08-22T17:00:55.441451401Z"
resource: {
labels: {
backend_service_name: ""
forwarding_rule_name: "lb-frontend-acme-protected"
project_id: "MY_PROJECT_ID"
target_proxy_name: ""
url_map_name: ""
zone: "global"
}
type: "http_load_balancer"
}
severity: "INFO"
timestamp: "2023-08-22T17:00:54.413317Z"
}

Why does this not work please?

If I change the clientValidationMode to ALLOW_INVALID_OR_MISSING_CLIENT_CERT it allows traffic through, so for some reason it's rejecting perfectly good client certificates which were issued by the Root CA (which is enabled).

UPDATE: I've also tried with cURL, but the connection is still reset, the same as the Go code: curl --key ~/.vipro-key.pem --cert ~/Downloads/client-cert.crt -v https://OUR_PROTECTED_ENDPOINT/