[SOLVED] Why can't ECS Fargate container find task execution role credentials?

Why can't ECS Fargate container find task execution role credentials?

We have Docker containers running in ECS Fargate. They use the AWS SDK for Go V2, and they set up the SDK like this

        cfg, err := config.LoadDefaultConfig(context.TODO())

We want to send an e-mail, so we set up SES:

        repo.client = ses.NewFromConfig(cfg)

When trying to send an e-mail, it cannot find credentials:

Get "http://169.254.169.254/latest/meta-data/iam/security-credentials/": dial tcp 169.254.169.254:80: connect: invalid argument

It seems to try to connect to the IMDS endpoint that belongs to ECS running on EC2, instead of the one for Fargate. What's going wrong here?

The full error for reference:

failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, exceeded maximum number of attempts, 3, request send failed, Get "http://169.254.169.254/latest/meta-data/iam/security-credentials/": dial tcp 169.254.169.254:80: connect: invalid argument

The ECS task execution role has a policy that allows full access to SES (for testing), so that's not the problem. The AWS documentation states that Fargate containers use the following address for credentials curl 169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html

Solution

The ECS Task Execution IAM role is used by the ECS service itself, for access to things like your ECR repository. This is the role used by ECS to access the other AWS services it needs to actually run your task.

The optional ECS Task IAM role is provided to your running ECS task containers to provide the software running in your containers access to AWS resources.

You need to provide your ECS task with a Task IAM role with the appropriate SES permissions. The AWS SDK you are using in your code will automatically pick up the ECS Task IAM role and use it.