Google Cloud Tasks cannot authenticate to Cloud Run

I am trying to invoke a Cloud Run service using Cloud Tasks as described in the docs here.

I have a running Cloud Run service. If I make the service publicly accessible, it behaves as expected.

I have created a cloud queue and I schedule the cloud task with a local script. This one is using my own account. The script looks like this

from google.cloud import tasks_v2

client = tasks_v2.CloudTasksClient()

project = 'my-project'
queue = 'my-queue'
location = 'europe-west1'
url = 'https://url_to_my_service'

parent = client.queue_path(project, location, queue)

task = {
        'http_request': {
            'http_method': 'GET',
            'url': url,
            'oidc_token': {
               'service_account_email': 'my-service-account@my-project.iam.gserviceaccount.com'
            }
        }
}

response = client.create_task(parent, task)
print('Created task {}'.format(response.name))

I see the task appear in the queue, but it fails and retries immediately. The reason for this (by checking the logs) is that the Cloud Run service returns a 401 response.

My own user has the roles "Service Account Token Creator" and "Service Account User". It doesn't have the "Cloud Tasks Enqueuer" explicitly, but since I am able to create the task in the queue, I guess I have inherited the required permissions. The service account "my-service-account@my-project.iam.gserviceaccount.com" (which I use in the task to get the OIDC token) has - amongst others - the following roles:

• Cloud Tasks Enqueuer (Although I don't think it needs this one as I'm creating the task with my own account)
• Cloud Tasks Task Runner
• Cloud Tasks Viewer
• Service Account Token Creator (I'm not sure whether this should be added to my own account - the one who schedules the task - or to the service account that should perform the call to Cloud Run)
• Service Account User (same here)
• Cloud Run Invoker

So I did a dirty trick: I created a key file for the service account, downloaded it locally and impersonated locally by adding an account to my gcloud config with the key file. Next, I run

curl -H "Authorization: Bearer $(gcloud auth print-identity-token)" https://url_to_my_service

That works! (By the way, it also works when I switch back to my own account)

Final tests: if I remove the oidc_token from the task when creating the task, I get a 403 response from Cloud Run! Not a 401... If I remove the "Cloud Run Invoker" role from the service account and try again locally with curl, I also get a 403 instead of a 401.

If I finally make the Cloud Run service publicly accessible, everything works.

So, it seems that the Cloud Task fails to generate a token for the service account to authenticate properly at the Cloud Run service.

What am I missing?

The next day I am no longer able to reproduce this issue. I can reproduce the 403 responses by removing the Cloud Run Invoker role, but I no longer get 401 responses with exactly the same code as yesterday. I guess this was a temporary issue on Google's side?

Also, I noticed that it takes some time before updated policies are actually in place (1 to 2 minutes).