amazon-web-serviceskubernetesaws-application-load-balancer

Target Group Binding not getting created for an ALB


While this question might seem duplicate at first, I would ask people to go through it once. I have checked SO, for all similar questions before posting this.

I have an ALB ingress controller which has a registered Target Group for an application that I am trying to access via the ALB. However the target group binding is not getting created for the application due to which the "registered targets" under the target group always comes as 0. Also the LoadBalancerAssociated also comes as None. This can be seen from the image below.

I have the checked the ALB pod logs and there is no error w.r.t creating the targetgroupbinding.

Based on some documentation here :

https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.1/guide/targetgroupbinding/targetgroupbinding/

I see that the ALB is supposed to create the targetgroupbinding itself:

The AWS LoadBalancer controller internally used TargetGroupBinding to support the functionality for Ingress and Service resource as well. It automatically creates TargetGroupBinding in the same namespace of the Service used.

Since there is no error in the pod logs, I am wondering how can I debug this issue?

enter image description hereAny help would be appreciated.

Update 1 (Current scenario) : The ALB is supposed to load balance a number of applications. So the ingress has many services under it. The targetgroupbindings have been created for all the service except the one mentioned above.


Solution

  • I seem to have figured out the solution to the issue.

    As mentioned in the question, the ALB ingress controller sits in front of a MANY services.

    Let's name them service A and service B, service B being the one having issues with target group binding.

    For service A there were below errors from the ALB logs:

    {"level":"info","ts":xxx.xx,"logger":"controllers.ingress","msg":"creating targetGroup","stackID":"xxxx","resourceID":"A"}
    {"level":"error","ts":xxxxxx.xxx,"logger":"controller-runtime.manager.controller.ingress","msg":"Reconciler error","name":"xxxxx","namespace":"xxxx","error":"InvalidParameter: 1 validation error(s) found.\n- minimum field value of 1, CreateTargetGroupInput.Port.\n"}
    

    The error suggested that ALB controller was unable to create a target group for Service A.

    However I ignored the error as it seemed unrelated to Service B.

    But, to my utter surprise, this error from the Reconciler seems to have been blocking the reconciliations to the other target groups.

    This was confirmed after fixing the above error by removing service A from the ALB Ingress yaml, which meant that the ALB would NOT create the Target group for service A.

    This led to the reconciliations for service B finally getting triggered :

    {"level":"info","ts":xxxx.xxx,"logger":"controllers.ingress","msg":"modifying targetGroupBinding","stackID":"xxx/xxxxx","resourceID":"xx/xxxx","targetGroupBinding":{"namespace":"xxxx","name":"xxxxxxx"}}
    {"level":"info","ts":xxxx.xxxxxxx,"logger":"controllers.ingress","msg":"modified targetGroupBinding","stackID":"xxx/xxxxxx","resourceID":"xxx/xxxxxx","targetGroupBinding":{"namespace":"xxxx","name":"xxxxxxx"}}
    

    And then eventually we had the target group for service B tagged with correct Load Balancer and targets.

    Most probable conclusion:

    Reconciler errors block all other reconciliations.So if you see that your target group bindings don't exist for an ALB, inspite of having the correct ingress configs and RBAC(update rights on targetgroupbinding CR), check for reconciler errors in the ALB pod logs.