amazon-web-servicesaws-cdkamazon-ecrvpc-endpoint

CannotPullContainerError in ECR when trying to use VPC Endpoints


I have an ECS stack made with AWS CDK.

import * as cdk from 'aws-cdk-lib'
import { Construct } from 'constructs'
import * as ec2 from 'aws-cdk-lib/aws-ec2'
import * as ecs from 'aws-cdk-lib/aws-ecs'
import path = require('path')

export class CdkPlaygroundEcsAuroraStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props)


    // VPC
    const vpc = new ec2.Vpc(this, 'ecs-vpc', {
      maxAzs: 2,
      subnetConfiguration: [
        {
          name: 'public',
          subnetType: cdk.aws_ec2.SubnetType.PUBLIC
        },
        {
          name: 'private',
          subnetType: cdk.aws_ec2.SubnetType.PRIVATE_ISOLATED
        }
      ]
    })


    // VPC Endpoints
    const ecrVpcEndpoint = new ec2.InterfaceVpcEndpoint(this, 'ECRVpcEndpoint', {
      vpc,
      service: ec2.InterfaceVpcEndpointAwsService.ECR,
      privateDnsEnabled: true
    })

    const cloudWatchVpcEndpoint = new ec2.InterfaceVpcEndpoint(this, 'cloudWatchVpcEndpoint', {
      vpc,
      service: ec2.InterfaceVpcEndpointAwsService.CLOUDWATCH,
      privateDnsEnabled: true
    })

    const s3GatewayEndpoint = new ec2.GatewayVpcEndpoint(this, 'S3GatewayEndpoint', {
      service: ec2.GatewayVpcEndpointAwsService.S3,
      vpc,
      subnets: [{ subnetType: cdk.aws_ec2.SubnetType.PRIVATE_ISOLATED }]
    })


    // ECS Cluster
    const ecsCluster = new ecs.Cluster(this, 'ecs-cluster', {
      vpc,
      enableFargateCapacityProviders: true,
      clusterName: 'ecs-cluster'
    })

    const taskDefinition = new ecs.FargateTaskDefinition(this, 'task1', {
      cpu: 256,
      memoryLimitMiB: 512
    })

    const container = taskDefinition.addContainer('container', {
      image: ecs.ContainerImage.fromAsset(path.resolve(__dirname, '../server')),
      memoryLimitMiB: 256
    })

    const service = new ecs.FargateService(this, 'server', {
      cluster: ecsCluster,
      taskDefinition,
      serviceName: 'server',
      vpcSubnets: {
        subnetType: cdk.aws_ec2.SubnetType.PRIVATE_ISOLATED
      }
    })
  }
}

It contains an ECS Task running in a private subnet without a NAT Gateway. I want to pull the image via the configured VPC Interface Endpoint. According to the documentation, I have setup VPC Endpoints for ECR, S3 and Cloudwatch.

But the task can not pull the image for the container with the following error:

CannotPullContainerError: pull image manifest has been retried 5 time(s): failed to resolve ref 650289367947.dkr.ecr.eu-central-1.amazonaws.com/cdk-hnb659fds-container-assets-650289367947-eu-central-1:d71b37a3ce0b63a08136ce5b816ea2d5d4b677fc8e3ee8b6eda9738d7c16ebb1: failed to do request: Head "https://650289367947.dkr.ecr.eu-central-1.amazonaws.com/v2/cdk-hnb659fds-container-assets-650289367947-eu-central-1/manifests/d71b37a3ce0b63a08136ce5b816ea2d5d4b677fc8e3ee8b6eda9738d7c16ebb1": dial tcp 3.121.190.14:443: i/o timeout

To me it looks like ECS is trying to access ECR over the internet (dial tcp 3.121.190.14:443). Why doesn't ECS use the VPC Interface Endpoint and how can I change this to fix the error?


Solution

  • To access ECR from ECS via VPC Endpoints, you require 3 endpoints. If you are using Fargate Version > 1.4.0, you will need the following endpoints:

    If you also wish to enable CloudWatch logging, you will additionally need the com.amazonaws.region.logs endpoint.

    The complete cdk definition would look like this:

    // access ECR
    new ec2.InterfaceVpcEndpoint(this, 'ECRVpcEndpoint', {
      vpc,
      service: ec2.InterfaceVpcEndpointAwsService.ECR,
      privateDnsEnabled: true
    })
    new ec2.InterfaceVpcEndpoint(this, 'ECRDockerVpcEndpoint', {
      vpc,
      service: ec2.InterfaceVpcEndpointAwsService.ECR_DOCKER,
      privateDnsEnabled: true
    })
    new ec2.GatewayVpcEndpoint(this, 'S3GatewayEndpoint', {
      service: ec2.GatewayVpcEndpointAwsService.S3,
      vpc,
      subnets: [{ subnetType: cdk.aws_ec2.SubnetType.PRIVATE_ISOLATED }]
    })
    
    // access Cloudwatch logging
    new ec2.InterfaceVpcEndpoint(this, 'CloudWatchLogsVpcEndpoint', {
      vpc,
      service: ec2.InterfaceVpcEndpointAwsService.CLOUDWATCH_LOGS,
      privateDnsEnabled: true
    })
    

    Docs