Error: ImagePullBackOff and Error: ErrImagePull errors with GKE

I'm getting the following error when kubectl apply a .yaml to deploy a custom built docker image to our cluster in GCP (edited out sensitive info):

Have tried the following but no luck:

• manually deploying the image
• checking to ensure there is firewall rules to allow 443 and there is, nothing blocking it either
• tried setting container registry to public
• checked service account perms and that the secrets for authorizing the pull is there
• verified the pull works on my end using the url given in the above image
• tried destroying whole infra. and respinning up
• ensured control planes are like for like as our other environment
• in our terraform, the cluster and node pools have all the required 0authscopes

Weird thing is, we have a dev. environment which is the exact same (using terraform to spin up the infra) and using the same files + images (obviously different projects so required things have been changed to match that) and no issues.

This happen usually when your cluster:

• has private nodes (aka no Public IP's)
• There is no Cloud NAT for the region of that cluster
• You don't have Private Access enabled on the subnet/vpc

Basically the error is that the node is not able to pull the image from the registry. You might have two cases:

• If your images are hosted in GCR or AR (Google Container Registry or Artifact Registry). Even if your nodes are private, Google Private Access must be enabled to allow the node to reach the registry
• If your images are hosted on an external registry (Exp: Docker). You have to have either nodes with a public IP or Cloud NAT to reach the internet
• The third case would be if you have self-hosted registries onPrem or on an other Cloud. You need some sort of private connection between GCP and where the registry is hosted (VPN, Interconnect...) and you need to configure Cloud DNS to resolve the registry to the private IP