I am trying to create a consul-api-gateway (Gateway) resource with HttpRoute for keycloak service in aws eks cluster. I am configuring in a way that the api-gateway and keycloak service are running in different namespaces. But when consul-api-gateway resource created with below deployment yaml, it is creating aws load balancer with the registered target showing as unhealthy. Not sure what makes this as unhealthy always.
Consul-server-and-ui-deployment.yaml
---
global:
name: consul
imagePullSecrets:
- name: "***"
image: /hashicorp/consul:1.12.0
imageEnvoy: /envoyproxy/envoy:v1.22.2
imageK8S: /hashicorp/consul-k8s-control-plane:0.44.0
metrics:
enabled: true
server:
replicas: 3
bootstrapExpect: 3
extraLabels:
tags.datadoghq.com/source: consul
tags.datadoghq.com/service: consul-server
annotations: |
"tags.datadoghq.com/source": "consul"
"tags.datadoghq.com/service": "consul-server"
updatePartitions: 3
disruptionBudget:
maxUnavailable: 1
tolerations: >
-
key: "taint_for_consul_xor_vault"
operator: "Equal"
value: "true"
effect: "NoSchedule"
nodeSelector: |
'eks.amazonaws.com/capacityType': 'ON_DEMAND'
'purpose': 'consul-server'
storage: 50G
priorityClassName: high-priority
client:
enabled: true
grpc: true
exposeGossipPorts: false
priorityClassName: high-priority
resources:
requests:
memory: 1G
cpu: "1"
limits:
memory: 1G
cpu: "1"
healthChecks:
enabled: true
hostNetwork: false
extraConfig: |
{
"advertise_reconnect_timeout": "15m",
"limits": {
"http_max_conns_per_client": -1
}
}
extraLabels:
tags.datadoghq.com/source: consul
tags.datadoghq.com/service: consul-client
annotations: |
"tags.datadoghq.com/source": "consul"
"tags.datadoghq.com/service": "consul-client"
updateStrategy: |
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
ui:
enabled: true
service:
type: LoadBalancer
additionalSpec: "'ports': [{'name': 'http', 'protocol': 'TCP', 'port': 8500,
'targetPort': 8500}, {'name': 'https', 'protocol': 'TCP', 'port': 8501,
'targetPort': 8501}]"
annotations: |
'service.beta.kubernetes.io/aws-load-balancer-scheme': 'internal'
'service.beta.kubernetes.io/aws-load-balancer-type': 'nlb-ip'
controller:
enabled: true
prometheus:
enabled: true
grafana:
enabled: true
terminatingGateways:
enabled: true
priorityClassName: high-priority
defaults:
replicas: 2
apiGateway:
enabled: true
image: /hashicorp/consul-api-gateway:0.3.0
controller:
replicas: 2
priorityClassName: high-priority
connectInject:
enabled: true
priorityClassName: high-priority
transparentProxy:
defaultEnabled: true
default: false
syncCatalog:
enabled: true
priorityClassName: high-priority
default: false
toConsul: true
toK8S: false
Consul-api-gateway-deployment.yaml
apiVersion: api-gateway.consul.hashicorp.com/v1alpha1
kind: GatewayClassConfig
metadata:
name: consul-common-gateway-class-config
spec:
logLevel: 'info'
copyAnnotations:
service:
- service.beta.kubernetes.io/aws-load-balancer-scheme
- service.beta.kubernetes.io/aws-load-balancer-type
consul:
scheme: 'http'
ports:
http: 8500
grpc: 8502
serviceType: LoadBalancer
---
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: GatewayClass
metadata:
name: consul-common-gateway-class
spec:
controllerName: 'hashicorp.com/consul-api-gateway-controller'
parametersRef:
group: api-gateway.consul.hashicorp.com
kind: GatewayClassConfig
name: consul-common-gateway-class-config
---
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: Gateway
metadata:
name: common-api-gateway
annotations:
'service.beta.kubernetes.io/aws-load-balancer-scheme': 'internal'
'service.beta.kubernetes.io/aws-load-balancer-type': 'nlb-ip'
spec:
gatewayClassName: consul-common-gateway-class
listeners:
- protocol: HTTP
port: 80
name: http
allowedRoutes:
namespaces:
from: All
HttpRoute-keycloak-deployment.yaml
---
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: HTTPRoute
metadata:
name: gateway-keycloak-route
namespace: app-ns
spec:
parentRefs:
- name: common-api-gateway
namespace: consul
rules:
- matches:
- path:
type: PathPrefix
value: /auth
backendRefs:
- kind: Service
name: keycloak
port: 80
namespace: app-ns
---
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: ReferencePolicy
metadata:
name: reference-policy-keycloak
spec:
from:
- group: gateway.networking.k8s.io
kind: HTTPRoute
namespace: app-ns
to:
- group: ""
kind: Service
name: keycloak
I wanted api-gateway load balancer running in aws eks to be with healthy registered targets and wanted to run Keycloak service have proper routes to api-gateway
I've had a very similar issue today, and spend hours trying to figure it out. I also had a very similar setup with a Consul API Gateway behind an internal AWS load balancer.
What solved it for me, was changing the load balancer type from ip to instance. In my case , I was using the AWS Load balancer controller, which seems to be the recommended way to go. So, I defined the annotations on Gateway like this like this:
service.beta.kubernetes.io/aws-load-balancer-type: "external"
service.beta.kubernetes.io/aws-load-balancer-scheme: "internal"
service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "instance"
And, of course, adding the service.beta.kubernetes.io/aws-load-balancer-nlb-target-type to the list of annotations to copy.
If you don't wish to install it, I think you can simply change the value of
'service.beta.kubernetes.io/aws-load-balancer-type from 'nlb-ip' to 'nlb'. I haven't tested it, but it should be the equivalent setting for the built-in kubernetes load balancer controller.
Hope this helps!
Also I have a few suggestion which are probably not related to the issue but might help you maintain a cleaner code.
You should change the deprecated ReferencePolicy object to a ReferenceGrant.
In addition, you might not need to define a custom GatewayClass and GatewayClassConfig, since you can control many of the settings of the load balancer through the values.yaml file you use to deploy consul, making your configuration shorter and simpler.
For example, this is the relevant part from my file
apiGateway:
enabled: true
image: "hashicorp/consul-api-gateway:0.5.1"
# Configuration settings for the default GatewayClass which consul will use.
managedGatewayClass:
serviceType: LoadBalancer
copyAnnotations:
service:
annotations: |
- service.beta.kubernetes.io/aws-load-balancer-type
- service.beta.kubernetes.io/aws-load-balancer-scheme
- service.beta.kubernetes.io/aws-load-balancer-nlb-target-type
And then in the Gateway I just set gatewayClassName: consul-api-gateway which is the default class.