kubernetesconsul

Error: K8s Failing to discover Consul server addresses

I am running a consul server outside of K8s. I have separatet k8s cluster and all nodes are reachable. I am not running any ACL or TLS at the moment. I am getting the below error and the injector pod is failing in K8s.

Consul server values:

data_dir = "/opt/consul"
client_addr = "0.0.0.0"

ui_config{
  enabled = true
}

server = true
advertise_addr = "192.168.60.10"
bootstrap_expect=1

ERROR:

2024-08-31T12:33:30.189Z [INFO]  consul-server-connection-manager.consul-server-connection-manager: trying to connect to a Consul server
2024-08-31T12:33:30.296Z [ERROR] consul-server-connection-manager.consul-server-connection-manager: connection error: error="failed to discover Consul server addresses: failed to resolve DNS name: consul-consul-server.consul.svc: lookup consul-consul-server.consul.svc on 10.96.0.10:53: no such host"

It seems even if I give the externalServer host IP it doesn’t work. Am I missing something here?

My helm values for k8s

global:
  enabled: false
  tls:
    enabled: false
  externalServers:
    enabled: true
    hosts: ["192.168.60.10"]
  server:
    enabled: false

I installed consul using helm

helm install consul hashicorp/consul --namespace consul -f helm-values.yaml

The resources in K8s

NAME                                                      READY   STATUS      RESTARTS   AGE
pod/consul-consul-connect-injector-bf57cf9b4-tzxcg        0/1     Running     0          30s
pod/consul-consul-gateway-resources-q44f7                 0/1     Completed   0          2m42s
pod/consul-consul-webhook-cert-manager-7c656f9967-hsr8v   1/1     Running     0          30s

NAME                                     TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)         AGE
service/consul-consul-connect-injector   ClusterIP      <none>        443/TCP         30s
service/consul-consul-dns                ClusterIP       <none>        53/TCP,53/UDP   30s

NAME                                                 READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/consul-consul-connect-injector       0/1     1            0           30s
deployment.apps/consul-consul-webhook-cert-manager   1/1     1            1           30s

NAME                                                            DESIRED   CURRENT   READY   AGE
replicaset.apps/consul-consul-connect-injector-bf57cf9b4        1         1         0       30s
replicaset.apps/consul-consul-webhook-cert-manager-7c656f9967   1         1         1       30s10.103.254.16610.97.215.246

When I check the logs in the inject pod it says below

k logs -n consul pod/consul-consul-connect-injector-bf57cf9b4-tzxcg

2024-08-31T12:33:30.189Z [INFO]  consul-server-connection-manager.consul-server-connection-manager: trying to connect to a Consul server
2024-08-31T12:33:30.296Z [ERROR] consul-server-connection-manager.consul-server-connection-manager: connection error: error="failed to discover Consul server addresses: failed to resolve DNS name: consul-consul-server.consul.svc: lookup consul-consul-server.consul.svc on 10.96.0.10:53: no such host"

I can ping to the consul server VM IP from K8s pod, also I could access services

curl http://192.168.60.10:8500/v1/catalog/services
{"consul":[]}

I sincerely appreciate if some one could kindly tell me what is wrong with this setup.

Thank you!

PS: I also checked the deployment consul-consul-connect-injector and it has below ENV variables

Environment:
      NAMESPACE:            (v1:metadata.namespace)
      POD_NAME:             (v1:metadata.name)
      CONSUL_ADDRESSES:    consul-consul-server.consul.svc
      CONSUL_GRPC_PORT:    8502
      CONSUL_HTTP_PORT:    8500
      CONSUL_DATACENTER:   dc1
      CONSUL_API_TIMEOUT:  5s
Solution

  • You should remove indentations of your helm-values.yaml. It should be:

    global:
  enabled: false
  tls:
    enabled: false
externalServers:
  enabled: true
  hosts: ["192.168.60.10"]
  httpsPort: 8500
server:
  enabled: false