K3OS can't pull image from AWS ECR private registry although it can push

I have found plenty of solutions for this problem all over the Internet, they all solve it using aws-cli and docker cli. Well, neither of them exist in K3OS. So I can't use them.

I created my image using Kaniko and successfully pushed it into a private ECR registry. For that purpose I created the configmap and secret as follows. Without using aws or docker cli.

kubectl create configmap docker-config --from-file=/home/rancher/.docker/config.json
kubectl create secret generic aws-secret --from-file=/home/rancher/.aws/credentials

However, when I used similar trick to pull the same image back from the same K3OS shell, I run out of luck. The pod.yaml is as follows:

apiVersion: v1
kind: Pod
metadata:
  name: tracker
spec:
  containers:
  - name: tracker
    image: xxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/tracker:latest
  imagePullSecrets:
  - name: aws-secret

I get the following error: Failed to pull image "xxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/tracker:latest": rpc error: code = Unknown desc = failed to pull and unpack image "xxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/tracker:latest": failed to resolve reference "xxxxxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/tracker:latest": pulling from host xxxxxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com failed with status code [manifests latest]: 401 Unauthorized

The IAM has the following policies enabled: AmazonEC2ContainerRegistryFullAccess EC2InstanceProfileForImageBuilderECRContainerBuilds AmazonElasticContainerRegistryPublicFullAccess

What am I missing here?

Solution

I finally solved it using the trick shown in this article.

Yes, it was a cop out. ;-) I ran aws cli in my windows PC.

aws ecr get-login-password --region us-east-1

And then copy-pasted the resulting password to the K3OS terminal in this command:

kubectl create secret docker-registry ecr-push-cred --docker-server=<your-registry-server> --docker-username=<your-name> --docker-password=<your-pword> --docker-email=<your-email>

Then used it in the pod definition:

  imagePullSecrets:
  - name: ecr-push-cred

I kept wondering why I could push the image, but not pull the image. I finally realized that while pushing, it was done in the container running the gcr.io/kaniko-project/executor. It could manage with secret of type generic. But K3OS required a secret of type docker-registry. So we need two types of secrets.