Getting 403 from Google Cloud Monitoring API using a service account and JWT auth

The code I'm running:

from google.oauth2 import service_account
from google.cloud import monitoring_v3

def build_time(isodate: str) -> int:
    from datetime import datetime
    
    utc_dt = datetime.strptime(isodate, '%Y-%m-%dT%H:%M:%S.%fZ')
    return int((utc_dt - datetime(1970, 1, 1)).total_seconds())

SCOPES = [
    "https://www.googleapis.com/auth/cloud-platform",
    "https://www.googleapis.com/auth/monitoring",
    "https://www.googleapis.com/auth/monitoring.read"
]
SERVICE_ACCOUNT_FILE = '/path/to/service_account_file.json'

credentials = service_account.Credentials.from_service_account_file(
        SERVICE_ACCOUNT_FILE, scopes=SCOPES)

client = monitoring_v3.MetricServiceClient(credentials=credentials)

name = 'projects/MY_PROJECT_NAME'
filter = 'metric.type = "loadbalancing.googleapis.com/l4_proxy/tcp/new_connections_count"'
end_time = build_time('2024-10-22T10:40:00.000Z')
start_time = build_time('2024-10-22T10:30:00.000Z')
interval = monitoring_v3.TimeInterval({
    "end_time": {"seconds": end_time},
    "start_time": {"seconds": start_time},
})
request = monitoring_v3.ListTimeSeriesRequest(
        name=name,
        filter=filter,
        interval=interval
    )

try:
    page_result = client.list_time_series(request=request)    
except Exception as error:
    print("Error happened:", error)
    exit()

print(page_result)

Dependencies:

google-auth==2.35.0
google-cloud-monitoring==2.23.0

I always get:

Error happened: 403 Permission monitoring.timeSeries.list denied (or the resource may not exist).

If I remove the credentials argument from MetricServiceClient, it falls back to my user account for authentication (configured with the gcloud cli tool) and it works. But when I use the JWT auth flow, configured for the service account, I always get the permission denied error. I followed the instructions here to set up the service account key.

EDIT: I'm running this on-prem. Only running locally for dev.

Please read the disclaimer on top of the page that you linked:

Important: If you are working with Google Cloud Platform, unless you plan to build your own client library, use service accounts and a Cloud Client Library instead of performing authorization explicitly as described in this document. For more information, see Authentication Overview in the Google Cloud Platform documentation.

Since you are using the client library you can use the regular service account authentication, no need to muck around yourself with oauth2 stuff: https://cloud.google.com/docs/authentication/provide-credentials-adc

If you are running this on-prem, you can do it similar to what you already did, just pass your SA json file location using the GOOGLE_APPLICATION_CREDENTIALS environment variable and use the default settings for the client. If it's not on prem you should never be using service account files, as mentioned in the link.

And obviously don't forget to give the appropriate role to the SA so that it has (among others) the monitoring.timeSeries.list permission. You probably want to use something like roles/monitoring.viewer, see https://cloud.google.com/iam/docs/understanding-roles#monitoring-roles for an overview