Pulling docker image in GKE

Apologies if this is a duplicate, I haven't found a solution in similar questions. I'm trying to upload a docker image to Google Kubernetes Engine. I've did it successfully before, but I can't seem to find my luck this time around.

I have Google SDK set up locally with kubectl and my Google Account, which is project owner and has all required permissions. When I use

kubectl create deployment hello-app --image=gcr.io/{project-id}/hello-app:v1

I see the deployment on my GKE console, consistently crashing as it "cannot pull the image from the repository.ErrImagePull Cannot pull image '' from the registry".

It provides 4 recommendations, which I have by now triple checked:

• Check for spelling mistakes in the image names.
• Check for errors when pulling the images manually (all fine in Cloud Shell)
• Check the image pull secret settings So, based on this https://blog.container-solutions.com/using-google-container-registry-with-kubernetes, I manually added 'gcr-json-key' from a new service account with project view permissions as well as 'gcr-access-token' to kubectl default service account.
• Check the firewall for the cluster to make sure the cluster can connect to the ''. Afaik, this should not be an issue with a newly set up cluster.

The pods themselve provide the following error code:

Failed to pull image "gcr.io/{project id}/hello-app:v1": 
[rpc error: code = Unknown desc = Error response from daemon: 
Get https://gcr.io/v2/{project id}/hello-app/manifests/v1: unknown: Unable to parse json key., 
rpc error: code = Unknown desc = Error response from daemon: 
Get https://gcr.io/v2/{project id}/hello-app/manifests/v1: 
unauthorized: Not Authorized., rpc error: code = Unknown desc = Error response from daemon: 
pull access denied for gcr.io/{project id}/hello-app, 
repository does not exist or may require 'docker login': denied: 
Permission denied for "v1" from request "/v2/{project id}/hello-app/manifests/v1".]

My question now, what am I doing wrong or how can I find out why my pods can't pull my image?

Kubernetes default serviceaccount spec:

kubectl get serviceaccount -o json
{
    "apiVersion": "v1",
    "imagePullSecrets": [
        {
            "name": "gcr-json-key"
        },
        {
            "name": "gcr-access-token"
        }
    ],
    "kind": "ServiceAccount",
    "metadata": {
        "creationTimestamp": "2020-11-25T15:49:16Z",
        "name": "default",
        "namespace": "default",
        "resourceVersion": "6835",
        "selfLink": "/api/v1/namespaces/default/serviceaccounts/default",
        "uid": "436bf59a-dc6e-49ec-aab6-0dac253e2ced"
    },
    "secrets": [
        {
            "name": "default-token-5v5fb"
        }
    ]
}

It does take several steps and the blog post you referenced appears to have them correctly. So, I suspect your error is in one of the steps.

Couple of things:

• The error message says Failed to pull image "gcr.io/{project id}/hello-app:v1". Did you edit the error message to remove your {project id}? If not, that's one problem.

• My next concern is the second line: Unable to parse json key. This suggests that you created the secret incorrectly:

1. Create the service account and generate a key
2. Create the Secret exactly as shown: kubectl create secret docker-registry gcr-json-key... (in the default namespace unless --namespace=... differs)
3. Update the Kubernetes spec with ImagePullSecrets

Because of the ImagePullSecrets requirement, I'm not aware of an alternative kubectl run equivalent but, you can try accessing your image using Docker from your host:

See: https://cloud.google.com/container-registry/docs/advanced-authentication#json-key

And then try docker pull gcr.io/{project id}/hello-app:v1 ensuring that {project id} is replaced with the correct GCP Project ID.

This proves:

• The Service Account & Key are correct
• The Container Image is correct

That leaves, your creation of the Secret and your Kubernetes spec to test.

NOTE The Service Account IAM permission of Project Viewer is overly broad for GCR access, see the permissions

Use StorageObject Viewer (roles/storage.objectViewer) if the Service Account needs only to pull images.