amazon-web-serviceskubernetesamazon-eksingress-controller

Service can't be registered in Target groups

I'm new to using Kubernetes and AWS so there are a lot of concepts I may not understand. I hope you can help me with this problem I am having.

I have 3 services, frontend, backend and auth each with their corresponding nodeport and an ingress that maps the one host to each service, everything is running on EKS and for the ingress deployment I am using AWS ingress controller. Once everything is deployed I try to register the node-group in the targets the frontend and auth services work correctly but backend stays in unhealthy state. I thought it could be a port problem but if you look at auth and backend they have almost the same deployment defined and both are api created with dotnet core. One thing to note is that I can do kubectl port-forward <backend-pod> 80:80 and it is running without problems. And when I run the kubectl describe ingresses command I get this:

Name:             ingress
Labels:           app.kubernetes.io/managed-by=Helm
Namespace:        default
Address:          xxxxxxxxxxxxxxxxxxxxxxxxxxx.xxxxx.elb.amazonaws.com
Ingress Class:    \<none\>
Default backend:  \<default\>
Rules:
Host                                  Path  Backends

----------------

domain.com
/   front-service:default-port (10.0.1.183:80,10.0.2.98:80)
back.domain.com
/   backend-service:default-port (\<none\>)
auth.domain.com
/   auth-service:default-port (10.0.1.30:80,10.0.1.33:80)
alb.ingress.kubernetes.io/listen-ports: \[{"HTTPS":443}, {"HTTP":80}\]
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/ssl-redirect: 443
kubernetes.io/ingress.class: alb

Events:
Type    Reason                  Age                   From     Message

-------------------------

Normal  SuccessfullyReconciled  8m20s (x15 over 41h)  ingress  Successfully reconciled

Frontend

apiVersion: apps/v1
kind: Deployment
metadata:
  name: front
  labels:
    name: front
spec:
  replicas: 2
  selector:
    matchLabels:
      name: front
  template:
    metadata:
      labels:
        name: front
    spec:
      containers:
      - name: frontend
        image:  {{ .Values.image }}
        imagePullPolicy: Always
        ports:
          - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: wrfront-{{ .Values.namespace }}-service
spec:
  type: NodePort
  ports:
  - port: 80
    targetPort: 80
    name: default-port
    protocol: TCP
  selector:
    name: front
---

Auth

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pvc-wrauth-keys
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 200Gi
---
apiVersion: "v1"
kind: "ConfigMap"
metadata:
  name: "auth-config-ocpm"
  labels:
    app: "auth"
data:
  ASPNETCORE_URL: "http://+:80"
  ASPNETCORE_ENVIRONMENT: "Development"
  ASPNETCORE_LOGGINGCONSOLEDISABLECOLORS: "true"
---
apiVersion: "apps/v1"
kind: "Deployment"
metadata:
  name: "auth"
  labels:
    app: "auth"
spec:
  replicas: 2
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: "auth"
  template:
    metadata:
      labels:
        app: "auth"
    spec:
      volumes:
        - name: auth-keys-storage
          persistentVolumeClaim:
            claimName: pvc-wrauth-keys
      containers:
        - name: "api-auth"
          image: {{ .Values.image }}
          imagePullPolicy: Always
          ports:
            - containerPort: 80
          volumeMounts:
            - name: auth-keys-storage
              mountPath: "/app/auth-keys"
          env:
            - name: "ASPNETCORE_URL"
              valueFrom:
                configMapKeyRef:
                  key: "ASPNETCORE_URL"
                  name: "auth-config-ocpm"
            - name: "ASPNETCORE_ENVIRONMENT"
              valueFrom:
                configMapKeyRef:
                  key: "ASPNETCORE_ENVIRONMENT"
                  name: "auth-config-ocpm"
            - name: "ASPNETCORE_LOGGINGCONSOLEDISABLECOLORS"
              valueFrom:
                configMapKeyRef:
                  key: "ASPNETCORE_LOGGINGCONSOLEDISABLECOLORS"
                  name: "auth-config-ocpm"
---
apiVersion: v1
kind: Service
metadata:
  name: auth-service
spec:
  type: NodePort
  selector:
    app: auth
  ports:
  - name: default-port
    protocol: TCP
    port: 80
    targetPort: 80

Backend (Service with problem)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: backend
  labels:
    app: backend
spec:
  replicas: 2
  selector:
    matchLabels:
      app: backend
  template:
    metadata:
      labels:
        app: backend
    spec:
      containers:
        - name: backend
          image: {{ .Values.image }}
          imagePullPolicy: Always
          ports:
            - containerPort: 80        
---
apiVersion: v1
kind: Service
metadata:
  name: backend-service
spec:
  type: NodePort
  selector:
    name: backend
  ports:
  - name: default-port
    protocol: TCP
    port: 80
    targetPort: 80

Ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    kubernetes.io/ingress.class: alb
    # SSL Settings
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}, {"HTTP":80}]'
    alb.ingress.kubernetes.io/ssl-redirect: '443'
    alb.ingress.kubernetes.io/certificate-arn: {{ .Values.certificate }}
spec:
  rules:
    - host: {{ .Values.host }}
      http:
        paths:
          - path: /
            backend:
              service:
                name: front-service
                port:
                  name: default-port
            pathType: Prefix
    - host: back.{{ .Values.host }}
      http:
        paths:
          - path: /
            backend:
              service:
                name: backend-service
                port:
                  name: default-port
            pathType: Prefix
    - host: auth.{{ .Values.host }}
      http:
        paths:
          - path: /
            backend:
              service:
                name: auth-service
                port:
                  name: default-port
            pathType: Prefix

I've tried to deploy other services and they work correctly, also running only backend or only another service, but always the same thing happens and always with the backend.

What could be happening? Is it a configuration problem? Some error in Ingress or Deployment? Or is it just the backend service?

I would be very grateful for any help.

Solution 
  • domain.com
/   front-service:default-port (10.0.1.183:80,10.0.2.98:80)
back.domain.com
/   backend-service:default-port (\<none\>)
auth.domain.com
/   auth-service:default-port (10.0.1.30:80,10.0.1.33:80)

    This one is saying that your backend service is not registered to the Ingress.

    One thing to remember is that Ingress registers Services by pods' ClusterIP, like your Ingress output "10.0.1.30:80", not NodePort. And according to docs , I don't know why you can have multiple NodePort services with the same port. But when you do port-forward, you actually open that port on all your instances, I assume you have 2 instances, and then your ALB health check that port and return healthy.

    But I think your issue is from your Ingress that can not locate your backend service.

    My suggestions are: