amazon-web-servicescontainersaws-cloudformationamazon-ecsecs-taskdefinition

Task definition contains multiple containers listening on different ports using one ECS service and ALB but some containers are not running


My Task definition contains 3 containers listening on different ports using single ECS service and a single ALB with different listener rules to diff containers. I am using cloudformation script to deploy the above. However only the default container(user) seems to be running and ALB routes all the path pattern to the default container. Sharing my cloud formation template below.

AWSTemplateFormatVersion: 2010-09-09
Description: >-
  This template is for Final Test of Devops


#   VPC
#   Internet Gateway
#   NAT Gateway
#   Public Route Table
#   Private Route Table
#   Public Subnets
#   Private Subnets
#   Security Groups
#   Application Load Balancer
#   Target Group

Parameters:
  EnvironmentName:
    Description: Stephen-vpc
    Type: String
    Default: Stephen-vpc

  VpcCIDR:
    Description: Please enter the IP range in CIDR notation (e.g., 10.0.0.0/16) for this VPC.
    Type: String
    Default: 10.0.0.0/16

  PublicSubnet1CIDR:
    Description: Please enter the IP range in CIDR notation (e.g., 10.0.10.0/24) for this public subnet in the first Availability Zone.
    Type: String
    Default: 10.0.10.0/24

  PublicSubnet2CIDR:
    Description: Please enter the IP range in CIDR notation (e.g., 10.0.11.0/24) for this public subnet in the second Availability Zone.
    Type: String
    Default: 10.0.11.0/24

  PrivateSubnet1CIDR:
    Description: Please enter the IP range in CIDR notation (e.g., 10.0.20.0/24) for this private subnet in the first Availability Zone.
    Type: String
    Default: 10.0.20.0/24

  PrivateSubnet2CIDR:
    Description: Please enter the IP range in CIDR notation (e.g., 10.0.21.0/24) for this private subnet in the second Availability Zone.
    Type: String
    Default: 10.0.21.0/24
  
  AvailabilityZone1:
    Description: Please select the availability zone for the subnet.
    Type: AWS::EC2::AvailabilityZone::Name
    Default: "ca-central-1a"

  AvailabilityZone2:
    Description: Please select the availability zone for the subnet.
    Type: AWS::EC2::AvailabilityZone::Name
    Default: "ca-central-1b"

  UsersDockerImage:
    Description: The Docker image for the ECS task.
    Type: String
    Default: <>
  PostsDockerImage:
    Description: The Docker image for the ECS task.
    Type: String
    Default: <>
  ThreadsDockerImage:
    Description: The Docker image for the ECS task.
    Type: String
    Default: <>

  ClusterName:
    Description: The name of the cluster.
    Type: String
    Default: DevopsCluster   

  UsersContainer:
    Description: The name of the container.
    Type: String
    Default: users

  PostsContainer:
    Description: The name of the container.
    Type: String
    Default: posts
  
  ThreadsContainer:
    Description: The name of the container.
    Type: String
    Default: threads

Resources:

  VPC:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: !Ref VpcCIDR
      EnableDnsSupport: true
      EnableDnsHostnames: true
      Tags:
        - Key: Name
          Value: !Sub ${EnvironmentName} VPC
      
  ## Internet Gateway

  InternetGateway:
    Type: AWS::EC2::InternetGateway
    Properties:
      Tags:
        - Key: Name
          Value: !Sub ${EnvironmentName} IG
  
  InternetGatewayAttachment:
    Type: AWS::EC2::VPCGatewayAttachment
    Properties:
      InternetGatewayId: !Ref InternetGateway
      VpcId: !Ref VPC

  ## Public Subnets
  
  PublicSubnet1:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VPC
      CidrBlock: !Ref PublicSubnet1CIDR
      AvailabilityZone: !Ref AvailabilityZone1
      Tags:
            - Key: Name
              Value: !Sub ${EnvironmentName} Pub-Sub-1

  PublicSubnet2:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VPC
      CidrBlock: !Ref PublicSubnet2CIDR
      AvailabilityZone: !Ref AvailabilityZone2
      Tags:
            - Key: Name
              Value: !Sub ${EnvironmentName} Pub-Sub-2

  ## Private Subnets

  PrivateSubnet1:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VPC
      CidrBlock: !Ref PrivateSubnet1CIDR
      AvailabilityZone: !Ref AvailabilityZone1
      Tags:
            - Key: Name
              Value: !Sub ${EnvironmentName} Pri-Sub-1

  PrivateSubnet2:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VPC
      CidrBlock: !Ref PrivateSubnet2CIDR
      AvailabilityZone: !Ref AvailabilityZone2
      Tags:
            - Key: Name
              Value: !Sub ${EnvironmentName} Pri-Sub-2

  ## NAT Gateway

  NATGateway1EIP:
    Type: AWS::EC2::EIP
    DependsOn: InternetGatewayAttachment
    Properties:
      Domain: VPC

  NATGateway2EIP:
    Type: AWS::EC2::EIP
    DependsOn: InternetGatewayAttachment
    Properties:
      Domain: VPC

  NATGateway1:
    Type: AWS::EC2::NatGateway
    Properties:
      AllocationId: !GetAtt NATGateway1EIP.AllocationId
      SubnetId: !Ref PublicSubnet1
      Tags:
            - Key: Name
              Value: !Sub ${EnvironmentName} NG1

  NATGateway2:
    Type: AWS::EC2::NatGateway
    Properties:
      AllocationId: !GetAtt NATGateway2EIP.AllocationId
      SubnetId: !Ref PublicSubnet2
      Tags:
            - Key: Name
              Value: !Sub ${EnvironmentName} NG2
  
  # Public Route Table

  PublicRouteTable:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref VPC
      Tags: 
            - Key: Name
              Value: !Sub ${EnvironmentName} Pub-RT
  
  PublicRoute:
    Type: AWS::EC2::Route
    DependsOn: InternetGatewayAttachment
    Properties:
      RouteTableId: !Ref PublicRouteTable
      DestinationCidrBlock: 0.0.0.0/0
      GatewayId: !Ref InternetGateway

  PublicSubnet1RouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      SubnetId: !Ref PublicSubnet1
      RouteTableId: !Ref PublicRouteTable

  PublicSubnet2RouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      SubnetId: !Ref PublicSubnet2
      RouteTableId: !Ref PublicRouteTable
  
  ## Private Route Table

  PrivateRouteTable1:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref VPC
      Tags: 
            - Key: Name
              Value: !Sub ${EnvironmentName} Pri-RT1

  PrivateRouteTable2:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref VPC
      Tags: 
            - Key: Name
              Value: !Sub ${EnvironmentName} Pri-RT2
              
  PrivateRoute1:
    Type: AWS::EC2::Route
    Properties:
      RouteTableId: !Ref PrivateRouteTable1
      DestinationCidrBlock: 0.0.0.0/0
      NatGatewayId: !Ref NATGateway1

  PrivateRoute2:
    Type: AWS::EC2::Route
    Properties:
      RouteTableId: !Ref PrivateRouteTable2
      DestinationCidrBlock: 0.0.0.0/0
      NatGatewayId: !Ref NATGateway2

  PrivateSubnet1RouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      SubnetId: !Ref PrivateSubnet1
      RouteTableId: !Ref PrivateRouteTable1

  PrivateSubnet2RouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      SubnetId: !Ref PrivateSubnet2
      RouteTableId: !Ref PrivateRouteTable2

############################################
# Resources section - Load Balancing
############################################

  ## Target Groups

  UsersTargetGroup:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties:
      VpcId: !Ref VPC
      Port: 3000
      Protocol: HTTP
      Matcher: 
        HttpCode: 200-299
      HealthCheckIntervalSeconds: 10
      HealthCheckPath: /
      HealthCheckProtocol: HTTP
      HealthCheckTimeoutSeconds: 5
      HealthyThresholdCount: 10
      TargetType: ip
  
  PostsTargetGroup:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties:
      VpcId: !Ref VPC
      Port: 3001
      Protocol: HTTP
      Matcher: 
        HttpCode: 200-299
      HealthCheckIntervalSeconds: 10
      HealthCheckPath: /
      HealthCheckProtocol: HTTP
      HealthCheckTimeoutSeconds: 5
      HealthyThresholdCount: 10
      TargetType: ip
  
  ThreadsTargetGroup:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties:
      VpcId: !Ref VPC
      Port: 3002
      Protocol: HTTP
      Matcher: 
        HttpCode: 200-299
      HealthCheckIntervalSeconds: 10
      HealthCheckPath: /
      HealthCheckProtocol: HTTP
      HealthCheckTimeoutSeconds: 5
      HealthyThresholdCount: 10
      TargetType: ip

  ## Application Load Balancer

  LoadBalancer:
    Type: AWS::ElasticLoadBalancingV2::LoadBalancer
    Properties:
      Name: 'app-elb'
      Subnets:
        - !Ref 'PublicSubnet1'
        - !Ref 'PublicSubnet2'
      SecurityGroups:
        - !Ref 'LoadBlancerSecurityGroup'

  ## Listener

  LoadBalancerListener:
    Type: AWS::ElasticLoadBalancingV2::Listener
    Properties:
      LoadBalancerArn: !Ref LoadBalancer
      Protocol: HTTP
      Port: 80
      DefaultActions:
        - Type: forward
          TargetGroupArn: !Ref UsersTargetGroup
  
  # ## Listener Rule

  ListenerRule1:
    Type: AWS::ElasticLoadBalancingV2::ListenerRule
    Properties:
      ListenerArn: !Ref LoadBalancerListener
      Priority: 2
      Conditions:
        - Field: path-pattern
          Values:
            - /api/users/*
      Actions:
        - TargetGroupArn: !Ref UsersTargetGroup
          Type: forward

  ListenerRule2:
    Type: AWS::ElasticLoadBalancingV2::ListenerRule
    Properties:
      ListenerArn: !Ref LoadBalancerListener
      Priority: 3
      Conditions:
        - Field: path-pattern
          Values:
            - /api/posts/*
      Actions:
        - TargetGroupArn: !Ref PostsTargetGroup
          Type: forward
  
  ListenerRule3:
    Type: AWS::ElasticLoadBalancingV2::ListenerRule
    Properties:
      ListenerArn: !Ref LoadBalancerListener
      Priority: 4
      Conditions:
        - Field: path-pattern
          Values:
            - /api/threads/*
      Actions:
        - TargetGroupArn: !Ref ThreadsTargetGroup
          Type: forward
  

############################################
# Resources section - ECS
############################################

  ## ECSCluster

  ECSCluster:
   Type: AWS::ECS::Cluster
   Properties:
    ClusterName: !Ref ClusterName

  ## Task

  Task:
    Type: AWS::ECS::TaskDefinition
    Properties:
      Family: 'users-task-def'
      Cpu: 1024
      Memory: 3072
      RuntimePlatform:  # Add RuntimePlatform specification
        CpuArchitecture: "X86_64"
        OperatingSystemFamily: "LINUX"
      NetworkMode: awsvpc
      RequiresCompatibilities: 
        - FARGATE
      ExecutionRoleArn: !Ref ECSTaskExecutionRole
      TaskRoleArn: !Ref ECSTaskExecutionRole
      ContainerDefinitions:
        - Name: !Ref UsersContainer
          Image: !Ref UsersDockerImage
          PortMappings:
            - ContainerPort: 3000
              Protocol: tcp
          LogConfiguration:
            LogDriver: awslogs
            Options:
              awslogs-group: '/ecs/users'  # Reference the existing log group
              awslogs-region: !Ref AWS::Region
              awslogs-stream-prefix: "ecs"
        - Name: !Ref PostsContainer
          Image: !Ref PostsDockerImage
          PortMappings:
            - ContainerPort: 3001
              Protocol: tcp
          LogConfiguration:
            LogDriver: awslogs
            Options:
              awslogs-group: '/ecs/posts'  # Reference the existing log group
              awslogs-region: !Ref AWS::Region
              awslogs-stream-prefix: "ecs"
        - Name: !Ref ThreadsContainer
          Image: !Ref ThreadsDockerImage
          PortMappings:
            - ContainerPort: 3002
              Protocol: tcp
          LogConfiguration:
            LogDriver: awslogs
            Options:
              awslogs-group: '/ecs/threads'  # Reference the existing log group
              awslogs-region: !Ref AWS::Region
              awslogs-stream-prefix: "ecs"

  ## Service

  Service:
    Type: AWS::ECS::Service
    DependsOn: 
      - ListenerRule1
      - ListenerRule2
      - ListenerRule3
    Properties:
      TaskDefinition: !Ref Task
      Cluster: !Ref ECSCluster
      LaunchType: FARGATE
      DesiredCount: 2
      DeploymentConfiguration:
        MaximumPercent: 200
        MinimumHealthyPercent: 70
      NetworkConfiguration:
        AwsvpcConfiguration:
          AssignPublicIp: DISABLED
          Subnets:
               - !Ref PrivateSubnet1
               - !Ref PrivateSubnet2
          SecurityGroups:
               - !Ref  ECSSecurityGroup
      LoadBalancers:
        - ContainerName: !Ref UsersContainer
          ContainerPort: 3000
          TargetGroupArn: !Ref UsersTargetGroup

############################################
# Resources section - Monitor and Log
############################################

  ## CloudWatchLogsGroup

  UserLogsGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: '/ecs/users'
      RetentionInDays: 1
  PostsLogsGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: '/ecs/posts'
      RetentionInDays: 1
  ThreadsLogsGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: '/ecs/threads'
      RetentionInDays: 1

############################################
# Resources section - Security
############################################

  ## Load Balancer Security Group
  LoadBlancerSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Allow external access for ALB
      VpcId: !Ref 'VPC'
      SecurityGroupIngress:
        - CidrIp: 0.0.0.0/0
          IpProtocol: TCP
          FromPort: 80
          ToPort: 80

  ## ECS Security Group
  ECSSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Allow traffic from ALB
      VpcId: !Ref 'VPC'
      SecurityGroupIngress:
        - SourceSecurityGroupId: !GetAtt LoadBlancerSecurityGroup.GroupId
          IpProtocol: TCP
          FromPort: 0
          ToPort: 65535

  ## ECSTaskExecutionRole

  ECSTaskExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: ECSTaskExecutionRole
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: ecs-tasks.amazonaws.com
            Action: sts:AssumeRole
      Path: /
      Policies:
        - PolicyName: ECSTaskExecutionRolePolicy
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  # ECS Tasks to download images from ECR
                  - 'ecr:GetAuthorizationToken'
                  - 'ecr:BatchCheckLayerAvailability'
                  - 'ecr:GetDownloadUrlForLayer'
                  - 'ecr:BatchGetImage'
                  # ECS Tasks to upload logs to CloudWatch
                  - 'logs:CreateLogStream'
                  - 'logs:PutLogEvents'
                Resource: '*'


Outputs:
  DNSName:
    Description: DNS name of the Application Load Balancer
    Value: !GetAtt LoadBalancer.DNSName

Solution

  • However only the default container(user) seems to be running

    You should look at the actual tasks/containers in the ECS web console, and the container logs in CoudWatch Logs to determine if they are running or not, instead of just guessing at that based on the behavior you are seeing. I think the containers are probably running fine, but you have some missing load balancer configuration.

    In your ECS Service definition, you are only registering the service with one target group. You need to change that to the following:

          LoadBalancers:
            - ContainerName: !Ref UsersContainer
              ContainerPort: 3000
              TargetGroupArn: !Ref UsersTargetGroup
            - ContainerName: !Ref PostsContainer
              ContainerPort: 3001
              TargetGroupArn: !Ref PostsTargetGroup
            - ContainerName: !Ref ThreadsContainer
              ContainerPort: 3002
              TargetGroupArn: !Ref ThreadsTargetGroup