aws sidecar container for sync s3 bucket to nginx is outputing 1000s of logs and stopping and starting

I have set up a task definition to run an nginx ecs container. Tested and works. I have added a sidecar container to share a volume with the nginx container, so it can upload content to the container as below.

I am getting huge amounts of verbose logs for the sidecar container and it just keeps starting and stopping my service. Any suggestions welcome. thanks

Note:

There are only 4 files in my s3 bucket. index.html, contact.html and some other html files.
All roles assigned to allow access to tasks.

task-definition.json

{
  "family": "nginx-example-development2",
  "requiresCompatibilities": [
    "EC2"
  ],
  "taskRoleArn": <ecsTaskRole>,
  "executionRoleArn": <ecsTaskExecutionRole>,
  "networkMode": "bridge",
  "containerDefinitions": [
      {
      "name": "s3-sync-container",
      "image": "amazon/aws-cli:latest",
      "memory": 256,
      "cpu": 128,
      "essential": true, 
      "command": [
        "/bin/sh", "-c", 
        "aws s3 sync  <S3_BUCKET> /shared/s3-cache --delete --only-show-errors && while true; do sleep 3600; done"
      ],
      "environment": [
        {
          "name": "AWS_DEFAULT_REGION",
          "value": <REGION>
        }
      ],
      "mountPoints": [
        {
          "sourceVolume": "shared-cache-volume",
          "containerPath": "/shared/s3-cache"
        }
      ],
      "healthCheck": {
        "command": ["CMD-SHELL", "test -f /usr/share/nginx/html/s3-cache/index.html && echo 'ready' || exit 1"],
        "interval": 30,
        "timeout": 5,
        "retries": 3,
        "startPeriod": 0
      },
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "my-sidecar-logs",
          "awslogs-region": "eu-west-2", 
          "awslogs-stream-prefix": "s3-sync"
        }
      }
    },
    {
      "name": "nginx-example-development2",
      "image":  <DOCKER_IMG>,
      "linuxParameters": {
        "initProcessEnabled": true
      },
      "cpu": 128,
      "memoryReservation": 256,
      "essential": true,
      "dependsOn": [
        {
          "containerName": "s3-sync-container",
          "condition": "HEALTHY"
        }
      ],
      "mountPoints": [
        {
          "sourceVolume": "shared-cache-volume",
          "containerPath": "/usr/share/nginx/html/s3-cache"
        }
      ],
      "environment": [
        {
          "name": "ECS_CONTAINER_METADATA_URI",
          "value": ""
        }, {
          "name": "NGINX_PORT",
          "value": "8080"
        }
      ],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-create-group": "true",
          "awslogs-group": "nginx-example-development2",
          "awslogs-region": <REGION>,
          "awslogs-stream-prefix": "ecs"
        }
      },
      "portMappings": [
        {
          "containerPort": 8080,
          "hostPort": 8080,
          "protocol": "tcp"
        }
      ],
      "healthCheck": {
        "command": [
          "CMD-SHELL",
          "/usr/local/bin/healthcheck.sh"
        ],
        "interval": 30,
        "timeout": 60,     
        "retries": 1, 
        "startPeriod": 30  
      },
      "command": [
        "nginx", "-g", "daemon off;"
      ]
    }
    
  ],
  "volumes": [
    {
      "name": "shared-cache-volume",
      "host": {
        "sourcePath": "/ecs/shared-cache"
      }
    }
  ]
}

These are the types of logs I'm getting: 1000s


08 October 2024 at 18:38 (UTC+1:00) iotthingsgraph | iottwinmaker uniqueid...
s3-sync-container

08 October 2024 at 18:38 (UTC+1:00) iotwireless | ivs uniqueid... s3-sync-container

08 October 2024 at 18:38 (UTC+1:00) ivs-realtime | ivschat uniqueid...
s3-sync-container

08 October 2024 at 18:38 (UTC+1:00)
kafka | kafkaconnect
uniqueid... s3-sync-container

08 October 2024 at 18:38 (UTC+1:00) kendra | kendra-ranking

Also I got this error but it hasn't indicated exactly what and its hard to filter so many logs:

aws: error: argument command: Invalid choice, valid choices are:

Solution

ok this was the problem.

Fix the verbose logs

"image": "amazon/aws-cli:latest", was causing the verbose messages. I'm not sure how to get around this. I decided to temporarily replaced it with: "image": "amazonlinux:2"

Fix config problems preventing containers from running healthy

After removing the verbose logs I could see the problems in the logs.

I had to fix a few config items:

the healthcheck should not be on nginx path but the volume path. ie: /shared/s3-cache/uploaded
I had to play around a little with the health check: startPeriod and timeout between the two tasks to allow it to finish.
I reduced the sleep to just 3 seconds to ensure it didn't run unnecessarily for an hr. Also added exit 0 after because I don't need it to run after the nginx starts.
I noticed my cmd was failing on the nginx container itself: nginx not a valid option. I just removed the cmd, since it was unnecessary. - "command": ["nginx", "-g", "daemon off;"]

task-definition-extract.json

{  
 "containerDefinitions": [{
    "image": "amazonlinux:2",
    "command": [
      "/bin/bash", "-c", 
      "touch /shared/s3-cache2/uploaded &&
       echo 'Files uploaded!' && 
       while true; do sleep 3; done && exit 0"
    ],
   "healthCheck": {
      "command": [
        "CMD-SHELL", 
       "test -f /shared/s3-cache/uploaded && echo 'ready' || exit 1"
      ],
    }
}

It worked.

Create new repo for sidecar to install aws-cli to run as sidecar I replaced that image with a new uploaded repo, that used the same "amazonlinux:2" and this time it installs the aws-cli on build, then runs the script to sync the files.

Dockerfile

FROM amazonlinux:2

RUN yum update -y && \
    yum install -y aws-cli && \
    mkdir -p /shared/s3-cache

COPY s3-sync-to-bucket.sh ./s3-sync-to-bucket.sh

ENTRYPOINT ["/bin/sh", "-c", "./s3-sync-to-bucket.sh"]

s3-sync-to-bucket.sh

#!/bin/sh
echo "syncing bucket"
aws s3 sync s3://$BUCKET_NAME /shared/s3-cache --delete --only-show-errors --quiet --exact-timestamps
touch /shared/s3-cache/uploaded
echo 'Files uploaded to bucket!'
while true; do sleep 3; done
exit 0

Updated task-definition.json


{
  "family": "nginx-example-development2",
  "requiresCompatibilities": [
    "EC2"
  ],
  "taskRoleArn": <ecsTaskRole>,
  "executionRoleArn": <ecsTaskExecutionRole>,
  "networkMode": "bridge",
  "containerDefinitions": [
      {
      "name": "nginx-example-dev-sidecar",
      "image": <side-car-image>,
      "memory": 256,
      "cpu": 128,
      "essential": true,
      "environment": [
        {
          "name": "AWS_DEFAULT_REGION",
          "value": "eu-west-2"
        }
      ],
      "mountPoints": [
        {
          "sourceVolume": "shared-cache-volume",
          "containerPath": "/shared/s3-cache"
        }
      ],
      "healthCheck": {
        "command": ["CMD-SHELL", "test -f /shared/s3-cache/uploaded && echo 'ready' || exit 1"],
        "interval": 60,
        "timeout": 10,
        "retries": 1,
        "startPeriod": 30
      },
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "nginx-example-dev-sidecar",
          "awslogs-region": <REGION>, 
          "awslogs-stream-prefix": "s3-sync"
        }
      }
    },
    {
      "name": "nginx-example-development2",
      "image": <nginx-image>,
      "linuxParameters": {
        "initProcessEnabled": true
      },
      "cpu": 128,
      "memoryReservation": 256,
      "essential": true,
      "dependsOn": [
        {
          "containerName": "nginx-example-dev-sidecar",
          "condition": "HEALTHY"
        }
      ],
      "mountPoints": [
        {
          "sourceVolume": "shared-cache-volume",
          "containerPath": "/usr/share/nginx/html/s3-cache"
        }
      ],
      "environment": [
        {
          "name": "ECS_CONTAINER_METADATA_URI",
          "value": ""
        }, {
          "name": "NGINX_PORT",
          "value": "8080"
        }
      ],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-create-group": "true",
          "awslogs-group": "nginx-example-development2",
          "awslogs-region": <REGION>,
          "awslogs-stream-prefix": "ecs"
        }
      },
      "portMappings": [
        {
          "containerPort": 8080,
          "hostPort": 8080,
          "protocol": "tcp"
        }
      ],
      "healthCheck": {
        "command": [
          "CMD-SHELL",
          "/usr/local/bin/healthcheck.sh"
        ],
        "interval": 30,
        "timeout": 60,     
        "retries": 1, 
        "startPeriod": 30  
      }
    }
    
  ],
  "volumes": [
    {
      "name": "shared-cache-volume",
      "host": {
        "sourcePath": "/ecs/shared-cache"
      }
    }
  ]
}

this now runs as expected.