gcloud.app.deploy Error Response: [13] Failed to create cloud build: invalid bucket

Suddenly, a few days ago gcloud app deploy for a Go app completely broke.

Steps to reproduce:

1. Enroll Google Cloud Platform for any gmail account and create a billing account there (there's a free trial).
2. Install Google Cloud CLI (including Beta) and do gcloud init / gcloud auth login <your_gmail>
3. Create a new project: gcloud projects create xxx-yyy
4. Enable billing for it: gcloud beta billing projects link xxx-yyy --billing-account XXX-YYY-XXX
5. Create App Engine for this project: gcloud app create --project xxx-yyy
6. Deploy the app: gcloud app deploy --appyaml myapp.yaml --project xxx-yyy
7. You will soon (after successfully uploading app's files) get this error:

ERROR: (gcloud.app.deploy) Error Response: [13] Failed to create cloud build: invalid bucket "205143635137.cloudbuild-logs.googleusercontent.com"; default Cloud Build service account or user-specified service account does not have access to the bucket
com.google.net.rpc3.client.RpcClientException: <eye3 title='/ArgoAdminNoCloudAudit.CreateBuild, INVALID_ARGUMENT'/> APPLICATION_ERROR;google.devtools.cloudbuild.v1/ArgoAdminNoCloudAudit.CreateBuild;invalid bucket "205143635137.cloudbuild-logs.googleusercontent.com"; default Cloud Build service account or user-specified service account does not have access to the bucket;AppErrorCode=3;StartTimeMs=1711390344614;unknown;ResFormat=uncompressed;ServerTimeSec=9.701026299;LogBytes=256;Non-FailFast;EndUserCredsRequested;EffSecLevel=none;ReqFormat=uncompressed;ReqID=a06ef84bab72e6f6;GlobalID=0;Server=[2002:a1b:7246:0:b0:32b:ca81:def]:4001.

Please note that it is completely irrelevant which account at step 1 you use. I used 4 (FOUR) different accounts where I have GCP enrolled (including the one where I have it enrolled for many years and where I created HUNDREDS of projects with apps deployed using the above commands) and the error is exactly the same all the time. Only the project number in the bucket's address is different of course.

Also note that everything still worked well on March, 22 (i.e. 3 days ago) — I have a couple of projects & apps deployed successfully on that day.

Also note that only new projects created after March, 22 are affected. For the older ones gcloud app deploy still works flawlessly.

I see that newer projects get different sets of default permissions in IAM, so Google definitely changed something sensitive there around March, 22. But I couldn't figure out which one is responsible for this problem — I tried to add all permissions the old projects have to the new ones and it still gives me the same error. I also tried all other solutions I could google (like enabling 'App Engine Admin' on the `Cloud Build Settings' page and doing gcloud auth login once again) — nothing helps.

It just looks like Google either broke something on their side 3 days ago or made some incompatible change without informing the users. From reading the error message, I could understand that the build process cannot access some internal bucket for writing Cloud Build logs and fails because of that. But since this bucket is hidden, I can't even check its permissions. I assume it should be properly and automatically set up by Google and this is exactly what Google Cloud docs say for the case when you use the default logging bucket (as I do). I'm lost.

PS. myapp.yaml is very trivial and I don't think that the problem has something to do with Go itself:

runtime: go121
app_engine_apis: true

main: mydomain/mypkg

env_variables:
  VAR: value

automatic_scaling:
  max_concurrent_requests: 80
  target_cpu_utilization: 0.6

handlers:
  - url: /.*
    script: auto
    secure: always

PPS. There is a similar question already but I created a new one as I have much more details to share.

This is a GCP outage. They offered a workaround that is working for me where you have to enable the Compute Engine API in the project you're trying to deploy to.