google-cloud-platformgoogle-cloud-runservice-accountsgoogle-cloud-tasks

Cloud Tasks cannot authenticate calls to Cloud Run Jobs

I've been racking my brain trying to figure out what's wrong with this setup. I've managed to get things going but think I'm just missing this last part.

I have a setup with the follow:

  1. Cloud Run Job A ("enqueue-tasks")
  2. Cloud Run Job B ("process-item")
  3. a Cloud Tasks queue ("process-items")

My goal is to schedule a biweekly trigger of the "enqueue-tasks" job. This job does some stuff (not important), and then queues up some tasks in the "process-items" queue.

This seems to be working fine

Each tasks should call the "process-item" job with some overrides. The queue should manage the rate-limit at which each tasks runs.

This seems to be the problem.

IAM Setup:

I created a custom role with the run.jobs.runWithOverrides permission so that I may pass additional overrides to the "process-item" job via args in each task

Excerpt of how I create/enqueue the tasks (note, everything is in us-west1 region):

// NOTE: This code is inside the "enqueue-tasks" job

import { CloudTasksClient } from "@google-cloud/tasks"

const client = new CloudTasksClient()

const project = "my-project-id"
const location = "us-west1"
const queueName = "process-items"
const taskId = "task-123" // example for item #123
const url = "https://run.googleapis.com/v2/projects/my-project-id/locations/us-west1/jobs/process-item:run"

await client.createTask({
    parent: client.queuePath(project, location, queueName),
    task: {
        name: client.taskPath(project, location, queueName, taskId),
        httpRequest: {
            httpMethod: "POST",
            url: url,
            oidcToken: {
                serviceAccountEmail: "cloud-task-service-account@my-project.iam.gserviceaccount.com",
                audience: url,
            },
            headers: {
                "Content-Type": "application/json",
            },
            body: Buffer.from(JSON.stringify({ overrides: { containerOverrides: [{args: "--item-id=123"}]}})).toString("base64"),
        },
    },
})

Now, even though I'm a little new to GCP, I believe I'm doing everything right! For some reason though, when the cloud tasks run, they always fails with status "UNAUTHORIZED" and when looking at the dashboard for my Cloud Run Jobs I don't see any invocation for them

To test and make sure that my cloud task SA is correct, I've gone ahead and created an access-token as the SA from the gcloud CLI with the following:

gcloud auth print-access-token --impersonate-service-account=cloud-task-service-account@my-project.iam.gserviceaccount.com

Then used Bruno (postman/insomnia like app) to try making the call manually as the cloud task SA.

> POST https://run.googleapis.com/v2/projects/my-project-id/locations/us-west1/jobs/process-item:run
> authorization: Bearer ya29.c.c0AY...XYZ
> content-type: application/json
> data {"overrides":{"containerOverrides":[{"args":["--item-id=123"]}]}}

This succeeds just fine! This leads me to believe that there must be something wrong with how im enqueueing the tasks. The docs are incredibly confusing and even digging through YouTube videos I can't seem to find any resources online that for from Cloud Run Job -> Cloud Tasks -> Cloud Run Job. Happy to add more details if needed but really curious if I'm just missing something here :/

After looking into this similar issue, I wonder if the issue is with the OIDC audience. I looked to the docs for the endpoint to trigger a cloud run job via REST and found this, but maybe there is another undocumented endpoint Im suppose to use with OIDC?

Solution

  • Google Cloud security is not so hard, but full of traps! And you felt in one of them. Let me explain.

    You must use OIDC (Identity Token) when you call YOUR api (deployed on Cloud Run, App Engine (with IAP) or Cloud Functions). You must use OAuth (access token) when you call Google APIs.

    You can notice that you use a print-access-token with your gcloud command and not a print-identity-token and it worked!

    Therefore, use a oauthToken (and without audience) instead of oidcToken in your task definition