I'm trying to create an autoscaling group manages EKS worker node provisioning. According to AWS' docs under the "Nodes fail to join cluster" section, in order for instances to join a cluster, the new instances must contain the tag kubernetes.io/cluster/my-cluster where my-cluster is the name of the cluster and the value of the tag must be owned. However, when the auto scaling group tries to provision new instances, I see the following error in the activity section:
Launching a new EC2 instance. Status Reason: Could not launch Spot Instances. InvalidParameterValue - 'kubernetes.io/cluster/my-cluster' is not a valid tag key. Tag keys must match pattern ([0-9a-zA-Z\-_+=,.@:]{1,255}), and must not be a reserved name ('.', '.', '_index'). Launching EC2 instance failed.
Does anyone know why this is happening and how I can address this?
I worked with AWS Support and discovered the issue is coming from a new feature called instance tags on EC2 instance metadata service.
This feature provides an alternative solution to making API calls via AWS CLI by allowing developers to use the metadata service API to query instance tags. This is useful to reduce the number of API calls if you are having issues with exceeding the maximum number of requests to AWS.
However, this causes conflicts with auto scaling group when the special IAM key is required which includes non-supported characters.
The solution to the problem is to set 'Metadata accessible' to 'Don't include in launch template' or 'Disabled' when creating your launch template.
You can find this option when creating or modifying a launch template under: Advanced details section > Metadata accessible