Vertex AI Workbench not running under specified service account for schedule-based recurring executions

I have a Workbench notebook that uses service account. It works when directly running through Workbench.

There are 2 ways to schedule an execution for async runs:

1. One-time execution
2. Schedule-based recurring execution

In both cases there is an advanced option to provide service account in which the script runs with default being

Use Vertex AI Training’s default service account

My service account is passed into execution for 'One-time execution' which can be verified from 'VIEW CUSTOM JOB INPUTS IN JSON'. Here's a snippet:

{
  "workerPoolSpecs": [
     ... bunch of config here ...
  ],
  "serviceAccount": "XXXXXXXXXX-compute@developer.gserviceaccount.com"
}

It executes fine.

'Schedule-based recurring execution', however, does not pass the service account even when specified and fails the script with PermissionDenied: 403 Request had insufficient authentication scopes. Here's the snippet from 'VIEW CUSTOM JOB INPUTS IN JSON' for schedule-based recurring execution with the missing 'serviceAccount' key

{
  "workerPoolSpecs": [
     ... bunch of config here ...
  ]
}

This appears to be a GCP bug unless I'm missing something.

To resolve the error of insufficient authentication scopes you can try including the authentication scope as "https://www.googleapis.com/auth/cloud-platform".

For example, you can include the below lines in your code.

from google.oauth2 import service_account

key_path = "path/to/service_account.json"  
credentials = service_account.Credentials.from_service_account_file(
    key_path,
    scopes=["https://www.googleapis.com/auth/cloud-platform"],)

Also, you can consider creating a custom service account as per your requirement.