amazon-web-servicesnginxamazon-ecsamazon-ecr

What is the right way to declare memory resource in AWS ECS task definition for Docker Nginx?


This is intended as a proof of concept example (with minimum setup).

Command to display resource uses by container instance:

instanceUID=$(aws ecs list-container-instances --cluster $project_name | jq -r '.containerInstanceArns[0]')
aws ecs describe-container-instances --cluster $project_name --container-instances $instanceUID | jq -r '.containerInstances[] | {remainingResources: .remainingResources, registeredResources: .registeredResources})'

Container instance memory usage output:

{
  "remainingResources": [
    {
      "name": "CPU",
      "type": "INTEGER",
      "doubleValue": 0.0,
      "longValue": 0,
      "integerValue": 1024
    },
    {
      "name": "MEMORY",
      "type": "INTEGER",
      "doubleValue": 0.0,
      "longValue": 0,
      "integerValue": 970
    },
...
  ],
  "registeredResources": [
    {
      "name": "CPU",
      "type": "INTEGER",
      "doubleValue": 0.0,
      "longValue": 0,
      "integerValue": 1024
    },
    {
      "name": "MEMORY",
      "type": "INTEGER",
      "doubleValue": 0.0,
      "longValue": 0,
      "integerValue": 970
    },
...
  ]
}

When I run a task against my task definition I get error: "RESOURCE:MEMORY":

aws ecs run-task \
  --cluster $project_name \
  --task-definition $project_name \
  --launch-type EC2
>  Troubleshoot error: {
    "tasks": [],
    "failures": [
        {
            "arn": "ARNID....",
            "reason": "RESOURCE:MEMORY"
        }
    ]
}

Example task definition:

{
  "family": "$project_name",
  "requiresCompatibilities": [
    "EC2"
  ],
  "executionRoleArn": "arn:aws:iam::$accountID:role/ecsTaskExecutionRole",
  "taskRoleArn": "arn:aws:iam::$accountID:role/ecs-tools-task-role",
  "networkMode": "bridge",
  "containerDefinitions": [
    {
      "name": "$project_name",
      "image": "$imageLocalTAG",
      "cpu": 256,
      "memoryReservation": 512,
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-create-group": "true",
          "awslogs-group": "$project_name",
          "awslogs-region": "$region",
          "awslogs-stream-prefix": "ecs"
        }
      },
      "portMappings": [
        {
          "hostPort": 80,
          "containerPort": 80,
          "protocol": "tcp"
        }
      ],
      "essential": true,
      "healthCheck": {
        "command": [
          "CMD-SHELL",
          "curl -f http://localhost/ || exit 1"
        ],
        "interval": 10,
        "retries": 3,
        "startPeriod": 0,
        "timeout": 5
      }
    }
  ]
}

This is a simple Docker Nginx application which renders hello world. Run locally docker stats only shows 4.8MIB (limit 3.8Gb) and max 10% CPU so I'm pretty sure there is plenty of memory declared in above task definition. The ECS optimized AMI using t2.micro should allow 1,024 CPU and 1GB memory.

How to fix this and run my instance successfully?


Solution

  • You are declaring in the task definition that you are using (requiring to be reserved for your task) 512MB of memory for each task. So each task requires half of the memory available on a t2.micro instance. Regardless of how much memory the tasks actually use, you are reserving 512MB for each task, which means nothing else on that server can use any of that memory, other than the ECS task that reserved it.

    The t2.micro instance also needs some memory to run the operating system, so it actually has somewhat less than 1GB of memory available to run ECS containers.

    That means that a t2.micro can only run one task that requires 512MB of memory. You need to reduce the memory requirements in the task definition if you want to run two of those tasks on a t2.micro server.