GCP Data Loss Prevention API Authentication: Does it require the use of a service account?

I am trying to automate DLP scans using the API. The only thing holding me back from finishing this project is authentication. It appears that creating and using a service account with BigQuery, Storage and DLP admin rights in each and every single project is the only way to avoid permission denied errors despite my own account having organization owner access to all projects. Is this an IAM issue or some requirement of DLP API and the use of tokens? Why does my token not work? We have a lot of projects and may end up with more in the future and would like to avoid the cumbersome and tedious issue of having to create a service account in each project especially having to do it again and again to keep the script working. Automation should be less work not more.

The error I get with my own account token below on a project were DLP API is definitely enabled.

"error": { "code": 403, "message": "Cloud Data Loss Prevention (DLP) API has not been used in project ###### before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/dlp.googleapis.com/overview?project=###### then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.", "status": "PERMISSION_DENIED", "details": [ { "@type": "type.googleapis.com/google.rpc.Help", "links": [ { "description": "Google developers console API activation", "url": "https://console.developers.google.com/apis/api/dlp.googleapis.com/overview?project=######"

Have a look at the documentation Authenticating to the Cloud DLP API:

You must authenticate to the Cloud DLP API in order to use it. The Cloud DLP API can handle both API keys and authentication. The main distinction between these two methods is:

• API keys identify the calling project—the app or site—that is making the call to an API.
• Auth tokens identify a user—the person—that is using the project.

and

To use a service account to authenticate to the Cloud DLP API:

Follow the instructions to create a service account. Select JSON as your key type, and grant the user the DLP User role (roles/dlp.user).

In general you should follow these steps:

1. enable billing
2. enable DLP API

3. create a service account, grant role to the service account roles/dlp.user and download the key.json file:

   $ gcloud iam service-accounts create test-dlp --description "test-cloud" --display-name "test-dlp"
   $ gcloud projects add-iam-policy-binding class-run --member serviceAccount:test-dlp@class-run.iam.gserviceaccount.com --role roles/dlp.user
   $ gcloud iam service-accounts keys create key.json --iam-account test-dlp@class-run.iam.gserviceaccount.com
   

4. set the env variable:

   export GOOGLE_APPLICATION_CREDENTIALS=[PATH_TO_key.json_FILE]
   

5. run your script

In addition, have a look at the Quickstart: Using the command-line tool section Permissions:

Inspecting content requires the serviceusage.services.use permission for the project that's specified in parent. The roles/editor, roles/owner, and roles.dlp.user roles contain the required permission or you can define your own custom role.

If you still have an issue, try to troubleshoot bu following these steps:

1. check your active account with command gcloud auth list
2. check if DLP API is enabled gcloud services list --enabled | grep DLP
3. activate your DLP service account with command gcloud auth activate-service-account and run your script again

and update your question with commands and outputs. Also, please clarify in which way do you use your DLP service account in your the script.