Kaniko on Gitlab CI fails uploading layer cache to Amazon ECR

Kaniko uploads the image to ECR but is unable to upload the layer cache.

What's wrong?
How can I fix it?

`.gitlab-ci.yml`

.backend:
  variables:
    AWS_PAGER: ""
    AWS_DEFAULT_REGION: eu-west-1
    RUNNER_IMAGE: ${ECR_REGISTRY}/${ECR_IMAGE}:latest
    RUNNER_IMAGE_COMMIT: ${ECR_REGISTRY}/${ECR_IMAGE}:${CI_COMMIT_SHORT_SHA}

  only:
    refs:
      - master
    changes:
      - acme/**/*
      - saleor/**/*
      - .gitlab-ci.yml
      - Dockerfile

Build:
  extends: .backend
  stage: build
  image:
    name: gcr.io/kaniko-project/executor:v1.9.1-debug
    entrypoint: [""]
  variables:
    AWS_ACCESS_KEY_ID: ${ECR_AWS_ACCESS_KEY_ID}
    AWS_SECRET_ACCESS_KEY: ${ECR_AWS_SECRET_ACCESS_KEY}
  script:
    - mkdir -p /kaniko/.docker
    - echo "{\"credsStore\":\"ecr-login\"}" > /kaniko/.docker/config.json
    - >-
      /kaniko/executor
      --cache=true
      --context $CI_PROJECT_DIR
      --dockerfile $CI_PROJECT_DIR/Dockerfile
      --destination $RUNNER_IMAGE
      --destination $RUNNER_IMAGE_COMMIT

Relevant lines from error Kaniko log:

INFO[0005] Taking snapshot of full filesystem...        
INFO[0007] Pushing layer xxxxxxxxxxxx.dkr.ecr.eu-west-1.amazonaws.com/acme-saleor-api/cache:85aca5582bb6057d98714aa75c1aee4e5646ecc08f8d6a412b135f0707578786 to cache now 
INFO[0007] Pushing image to xxxxxxxxxxxx.dkr.ecr.eu-west-1.amazonaws.com/acme-saleor-api/cache:85aca5582bb6057d98714aa75c1aee4e5646ecc08f8d6a412b135f0707578786 
WARN[0013] Error uploading layer to cache: failed to push to destination xxxxxxxxxxxx.dkr.ecr.eu-west-1.amazonaws.com/acme-saleor-api/cache:da1a715b9dc68826590c35ad9948cebd8b0c52cdb4b354ef3d554411de3c1102: Post "https://xxxxxxxxxxxx.dkr.ecr.eu-west-1.amazonaws.com/v2/acme-saleor-api/cache/blobs/uploads/": EOF

ECR Policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "ecr:BatchCheckLayerAvailability",
                "ecr:BatchDeleteImage",
                "ecr:BatchGetImage",
                "ecr:BatchImportUpstreamImage",
                "ecr:CompleteLayerUpload",
                "ecr:CreatePullThroughCacheRule",
                "ecr:CreateRepository",
                "ecr:DescribeImages",
                "ecr:DescribeRepositories",
                "ecr:GetAuthorizationToken",
                "ecr:GetDownloadUrlForLayer",
                "ecr:GetRepositoryPolicy",
                "ecr:InitiateLayerUpload",
                "ecr:ListImages",
                "ecr:PutImage",
                "ecr:PutImageScanningConfiguration",
                "ecr:PutImageTagMutability",
                "ecr:PutLifecyclePolicy",
                "ecr:PutRegistryScanningConfiguration",
                "ecr:PutReplicationConfiguration",
                "ecr:ReplicateImage",
                "ecr:StartImageScan",
                "ecr:StartLifecyclePolicyPreview",
                "ecr:TagResource",
                "ecr:UntagResource",
                "ecr:UploadLayerPart"
            ],
            "Resource": "*"
        }
    ]
}

Solution

I'm assuming that ECR wasn't happy with putting the cache in a "subdirectory" because as soon as I explicitly set the --cache-repo argument, this started working.

.backend:
  variables:
    AWS_PAGER: ""
    AWS_DEFAULT_REGION: eu-west-1
    RUNNER_IMAGE: ${ECR_REGISTRY}/${ECR_IMAGE}:latest
    RUNNER_IMAGE_COMMIT: ${ECR_REGISTRY}/${ECR_IMAGE}:${CI_COMMIT_SHORT_SHA}
    CACHE_TTL: 2190h0m0s # three months
  # ...

Build:
  extends: .backend
  stage: build
  image:
    name: gcr.io/kaniko-project/executor:v1.9.1-debug
    entrypoint: [""]
  variables:
    AWS_ACCESS_KEY_ID: ${ECR_AWS_ACCESS_KEY_ID}
    AWS_SECRET_ACCESS_KEY: ${ECR_AWS_SECRET_ACCESS_KEY}
  script:
    - mkdir -p /kaniko/.docker
    - echo "{\"credsStore\":\"ecr-login\"}" > /kaniko/.docker/config.json
    - >-
      /kaniko/executor
      --context $CI_PROJECT_DIR
      --dockerfile $CI_PROJECT_DIR/Dockerfile
      --destination $RUNNER_IMAGE
      --destination $RUNNER_IMAGE_COMMIT
      --cache=true
      --cache-repo ${ECR_REGISTRY}/${ECR_IMAGE}
      --cache-ttl $CACHE_TTL