google-cloud-platformgoogle-cloud-rungoogle-cloud-run-jobs

Previously successful Cloud Run Service & Job deploy process with resource level IAM grants not working anymore for certain projects

I have service account example@project-a.

In project-b, I have cloud run job example, for which I have granted resource level IAM binding:

gcloud run jobs get-iam-policy --project project-b example

bindings:
- members:
  - serviceAccount:example@project-a.iam.gserviceaccount.com
  role: roles/run.developer

Prior to August 5, the following worked:

gcloud run jobs update example
    --region=example-region
    --project=project-b
    --image=example-image

Then suddenly, the deploy stopped working. I have not changed anything.

ERROR: (gcloud.run.jobs.update) PERMISSION_DENIED: Permission 'run.jobs.update' denied on resource (or it may not exist). 
This command is authenticated as example@project-a.iam.gserviceaccount.com

Same error exists with both gcloud cli v486 and v484.

And also, I have a pre-production project project-c, where an identical deploy still works.

I have verified that seemingly there is no org level or folder level policy, which should interfere deploy.

Also, running gcloud policy-troubleshoot iam --project project-b //run.googleapis.com/projects/project-b/locations/example-region/jobs/example --principal-email=example@project-a.iam.gserviceaccount.com --permission=run.jobs.update tells me:

- access: GRANTED
  bindingExplanations:
  - access: GRANTED
    memberships:
      serviceAccount:example@project-a.iam.gserviceaccount.com:
        membership: MEMBERSHIP_INCLUDED
        relevance: HIGH
    relevance: HIGH
    role: roles/run.developer
    rolePermission: ROLE_PERMISSION_INCLUDED
    rolePermissionRelevance: HIGH

EDIT: as of 9 Aug 06:00 UTC Cloud Run deploys also stopped working in europe region for both preproduction and production projects.

Solution

  • When I add project level binding of Cloud Run Developer to example@project-a in project-b, then deploys started working again.

    BUT, it should also work with resource level bindings ...

    EDIT: This has now been fixed internally by Google. Our setup started working again without any changes.