dockerkubernetesgoogle-cloud-platformoauthauthorization

Error: ErrImagePull : failed to fetch oauth token: unexpected status: 403 Forbidden while creating kubernetes deployment on Google Cloud

Case

I am following tutorial: Deploy a Spring Boot Java app to Kubernetes on Google Kubernetes Engine . I came to the step when I want to deploy my application (docker image of it) to kubernetes. But after I create the deployment the pods are not starting because there is problem with pulling the image (Error: ErrImagePull).

I create the deployment like that:

 kubectl create deployment questy-java --image=us-central1-docker.pkg.dev/quizdev/codelabrepo/questy-java:v1

After running the command:

kubectl describe pod questy-java-54dbd6ccd4-5cb94

I am getting event information:

Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  36s                default-scheduler  Successfully assigned default/questy-java-54dbd6ccd4-tmdkl to gke-questy-java-cluster-default-pool-ca7ad417-lxkw
  Normal   Pulling    23s (x2 over 35s)  kubelet            Pulling image "us-central1-docker.pkg.dev/quizdev/codelabrepo/questy-java:v1"
  Warning  Failed     23s (x2 over 35s)  kubelet            Failed to pull image "us-central1-docker.pkg.dev/quizdev/codelabrepo/questy-java:v1": rpc error: code = Unknown desc = failed to pull and unpack image "us-central1-docker.pkg.dev/quizdev/codelabrepo/questy-java:v1": failed to resolve reference "us-central1-docker.pkg.dev/quizdev/codelabrepo/questy-java:v1": failed to authorize: failed to fetch oauth token: unexpected status: 403 Forbidden
  Warning  Failed     23s (x2 over 35s)  kubelet            Error: ErrImagePull
  Normal   BackOff    8s (x2 over 35s)   kubelet            Back-off pulling image "us-central1-docker.pkg.dev/quizdev/codelabrepo/questy-java:v1"
  Warning  Failed     8s (x2 over 35s)   kubelet            Error: ImagePullBackOff

The root couse seems to be mentioned in here:

failed to authorize: failed to fetch oauth token: unexpected status: 403 Forbidden

Anyhow the tutorial did not mention any additional steps to mantain authentication or authorisation.

What I have tried:

  1. I have checked the VPC networks dashboard to see if Private Google Access is enabled and it looks fine:

enter image description here

  1. I have followed instructions to install gke-gcloud-auth-plugin from: https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke. Recreating the deployment after this change did not help.
  2. Created new repository in the Artifact Registry and tried to deploy my image there
  3. build an entirely new Docker container and tried to deploy it in new repository
  4. Tried to deploy questy-java image directly from Google Cloud Console: enter image description here
  1. Run the command:
    gcloud auth configure-docker us-central1-docker.pkg.dev
  1. Checked the path throughly. I have copied and paste the path to the image directly from Console: enter image description here I have also tried to run the commands:
    docker pull \
        us-central1-docker.pkg.dev/quizdev/codelabrepo/questy-java:v1

It was successfull.

Question

What could be the reason of this error? What is the most elegant way to fix the issue?

Edit

I have checked the path to the image directly from Console and it is

  us-central1-docker.pkg.dev/quizdev/codelabrepo/questy-java:v1
Solution

  • The Error: ErrImagePull : failed to fetch oauth token: unexpected status: 403 Forbidden is due to authorization issue where the Artifact registry is not having the necessary permissions to pull the image.

    Refer to this Troubleshooting error 4xx issues:

    Authentication and authorization errors when connecting to GKE clusters. Set the environment variables to print the access token and

    Verify that your access token is valid by following the steps mentioned in the Troubleshooting error 4xx issues.

    Also as per this Troubleshoot doc you can check the following:

    • Verify that the full path of the image that you are pushing is correct. The path must include the registry hostname, Google Cloud
      project ID, repository, and image. For example:
      us-east1-docker.pkg.dev/my-project/my-rep
    • Verify that the account that is pulling the image has necessary permissions to read from the repository. you must grant the Artifact Registry Reader role to the runtime service account then only
      this will help to pull the image.
    • If you are using Docker or another third-party tool, you must: Grant permissions to the account that interacts with the
      repository.