Hashicorp Consul - How to do verified TLS from Pods in Kubernetes cluster

I'm having some difficulty understanding Consul end-to-end TLS. For reference, I'm using Consul in Kubernetes (via the hashicorp/consul Helm chart). Only one datacenter and Kubernetes cluster - no external parties or concerns.

I have configured my override values.yaml file like so:

global:
  datacenter: sandbox

  gossipEncryption:
    secretName: "consul"
    secretKey: "CONSUL_GOSSIP_ENCRYPTION_KEY"

  tls:
    enabled: true
    httpsOnly: true
    enableAutoEncrypt: true
    serverAdditionalDNSSANs: ["'consul.service.consul'"]

server:
  replicas: 3
  bootstrapExpect: 3
  storage: 20Gi

dns:
  clusterIP: 172.20.53.53

ui:
  service:
    type: 'LoadBalancer'

syncCatalog:
  enabled: true

All other values are as default from the shipped values.yaml file.

This works, and Consul client logs suggest that all agents area connecting nicely using TLS, with relevant certs and keys being created by (as I understand) the Auto-encryption feature of Consul.

What I don't understand is how to initiate a HTTPS connection from an application on Kubernetes, running in a Pod, to a Consul server. Since the Pod's container does not (presumably) have the Consul root CA cert in its trust store, all HTTPS calls fail, as per wget example below:

# Connect to Pod:
laptop$> kubectl exec -it my-pod sh

# Attempt valid HTTPS connection:
my-pod$> wget -q -O - https://consul.service.consul:8501
Connecting to consul.service.consul:8501 (10.110.1.131:8501)
ssl_client: consul.service.consul: certificate verification failed: unable to get local issuer certificate
wget: error getting response: Connection reset by peer

# Retry, but ignore certificate validity issues:
my-pod$> wget --no-check-certificate -q -O - https://consul.service.consul:8501/v1/status/leader
"10.110.1.131:8300"

How am I supposed to enforce end-to-end (verified) HTTPS connections from my apps on Kubernetes to Consul if the container does not recognize the certificate as valid? Am I misunderstanding something about certificate propagation?

Many thanks - Aaron

Solution

Solved with thanks to Hashicorp on their Consul discussion forum.

Create a Kubernetes secret named consul with a key named CONSUL_GOSSIP_ENCRYPTION_KEY and an appropriate encryption key value.
- Generate value using consul keygen
Install the hashicorp/consul Helm chart with an values-override.yaml , such as below:

global:
  datacenter: sandbox

  gossipEncryption:
    secretName: "consul"
    secretKey: "CONSUL_GOSSIP_ENCRYPTION_KEY"

  tls:
    enabled: true
    httpsOnly: true
    enableAutoEncrypt: true
    serverAdditionalDNSSANs: ["'consul.service.consul'"]

server:
  replicas: 3
  bootstrapExpect: 3
  storage: 20Gi

dns:
  clusterIP: 172.20.53.53

ui:
  service:
    type: 'LoadBalancer'

syncCatalog:
  enabled: true

Create an example Pod spec to represent our application.
- Ensure it mounts the Consul server CA cert secret.
- Ensure the Pod’s container has HOST_IP exposed as an environment variable.

apiVersion: v1
kind: Pod
metadata:
  namespace: default
  name: test-pod
spec:
  volumes:
  - name: consul-consul-ca-cert
    secret:
      secretName: consul-consul-ca-cert
  hostNetwork: false
  containers:
  - name: consul-test-pod
    image: alpine
    imagePullPolicy: IfNotPresent
    env:
    - name: HOST_IP
      valueFrom:
        fieldRef:
          fieldPath: status.hostIP
    command: ["/bin/sh"]
    args: ["-c", "while true; do sleep 24h; done"]
    volumeMounts:
    - name: consul-consul-ca-cert
      mountPath: /consul/tls/ca

Upon creation of the Pod, kubectl exec into it, and ensure the ca-certificates and curl packages are installed (I’m using Alpine Linux in this example).
- (curl is purely for testing purposes)

#> apk update
#> apk add ca-certificates curl

Copy the mounted Consul server CA certificate into the /usr/local/share/ca-certificates/ and execute update-ca-certificates to add it to the system root CA store.

#> cp /consul/tls/ca/tls.crt /usr/local/share/ca-certificates/consul-server-ca.crt
#> update-ca-certificates  # might give a trivial warning - ignore it

The Consul server is now accessible (and trusted) over HTTPS as below:

#> curl https://consul.service.consul:8501/v1/status/leader
## No TLS errors ##

We also want to talk to the Consul client (instead of the server) over HTTPS, for performance reasons.
- Since the Consul client has its own CA cert, we need to retrieve that from the server.
- This requires the consul-k8s binary, so we need to get that.

#> cd /usr/local/bin
#> wget https://releases.hashicorp.com/consul-k8s/0.15.0/consul-k8s_0.15.0_linux_amd64.zip  # (or whatever latest version is)
#> unzip consul-k8s_0.15.0_linux_amd64.zip
#> rm consul-k8s_0.15.0_linux_amd64.zip

Get the Consul client CA cert and install it via update-ca-certificates :

#> consul-k8s get-consul-client-ca -server-addr consul.service.consul -server-port 8501 -output-file /usr/local/share/ca-certificates/consul-client-ca.crt
#> update-ca-certificates  # might give a trivial warning - ignore it

The Consul client is now accessible (and trusted) over HTTPS as below:

#> curl https://$HOST_IP:8501/v1/status/leader
## No TLS errors ##

We can also access the Consul KV service from the client without issue:

#> curl https://$HOST_IP:8501/v1/kv/foo/bar/baz
## No TLS errors ##

Naturally, all of the above should be automated by the implementer. These manual steps are purely for demonstration purposes.