[SOLVED] gRPC server running on EKS cluster accepts traffic when TLS is disabled, but refuses connection when using certificates created by awspca/cert-manager

gRPC server running on EKS cluster accepts traffic when TLS is disabled, but refuses connection when using certificates created by awspca/cert-manager

I wrote this sample grpc server : https://github.com/chasehippen/grpc-server-test (some values removed for sensitivity) where I've tried to set up a tls server that will do client authentication using a set of certs that I've generated with a cert-manager "Certificate" resource. I'm trying to use the tls.crt, tls.key, and ca.crt files in my server and in my client.

I'm essentially trying to get grpc traffic working over tls, but I'm having trouble.

When I set the env

env:
  - name: SERVER_TLS_ENABLED
    value: "true"

and I have the istio destinationrule and peerauthentication setup, I get

❯ grpcurl -proto ../grpc-server/proto/greeter.proto -d '{ "name": "Chase" }' grpc-server.dev-cluster.xxx.xxxx.xx.com:31111 example.Greeter/SayHello
ERROR:
  Code: Unavailable
  Message: upstream connect error or disconnect/reset before headers. retried and the latest reset reason: remote connection failure, transport failure reason: TLS error: Secret is not supplied by SDS

When the env is set to true, but peerauthentication and destinationrule are deleted I get:

❯ grpcurl -proto ../grpc-server/proto/greeter.proto -d '{ "name": "Chase" }' -cert="tls.crt" -key="tls.key" grpc-server.dev-cluster.xxx.xxxx.xx.com:31111 example.Greeter/SayHello
ERROR:
  Code: Unavailable
  Message: upstream connect error or disconnect/reset before headers. reset reason: connection termination

With all that set, when I actually set the cacert I get:

❯ grpcurl -proto ../grpc-server/proto/greeter.proto -d '{ "name": "Chase" }' -cacert="ca.crt" -cert="tls.crt" -key="tls.key" grpc-server.dev-cluster.xxx.xxxx.xx.com:31111 example.Greeter/SayHello
Failed to dial target host "grpc-server.dev-cluster.xxx.xxxxx.xx.com:31111": tls: failed to verify certificate: x509: certificate signed by unknown authority

When the env is set to:

env:
- name: SERVER_TLS_ENABLED
  value: "false"

then I get:

{
  "message": "Hello Chase"
}

So the grpc calls from client to server work but only when TLS is turned off. I can't figure out where the mismatch is between the certs I'm generating with cert-manager.

Also I get the same results when I have istio-proxy sidecars enabled on the namespace.

I also commented out the grpc_health_probe readiness and liveness probe because they were failing with TLS enabled. They succeed when TLS is disabled.

Solution

Figured out the issue was I needed to set the common name and dns names to the service name, and that resolved it.