Google Cloud Platform Secret Manager API 403 Error

In our project that runs in the Google Cloud Platform (GCP) we are seeing the following HTTP 403 error in the Google Cloud Console / APIs & Services:

google.cloud.secretmanager.v1.SecretManagerService.AccessSecretVersion

I'm trying to figure out what exactly is causing the error, but I can not find anything in the Google Cloud Console / Log Explorer. I tried using this query:

resource.type="audited_resource"
protoPayload.serviceName="secretmanager.googleapis.com"
protoPayload.status.code>=400

I also tried this via the termin:

gcloud logging read 'protoPayload.status.code=403' --project="my-project" --format="json"

Now I'm running out of ideas. The problem occured first on August 8th, 2024 around 6:00a.m. Central European Summer Time (CEST).

I'm grateful for any idea!

In order to query Secret Manager you may need to enable Data Access audit logs for Secret Manager.

See Secret Manager Audit Logging which explains which permission types gets which service methods.

However please note: when I tried this, my data access audit logs don't include authenticationInfo (and so no principalEmail). This may be a requirement|consequence of Data Access audit logs with Secret Manager service (I'm not sure) and so, enabling these logs may be helpful but not conclusive as you won't know who|what made the erroneous calls.

Browse: https://console.cloud.google.com/iam-admin/audit?project=${PROJECT}

And enable "Admin Read", "Data Read" and|or "Data Write"

You should then see cloudaudit.googleapis.com%2Fdata_access added to your logs:

gcloud logging list --project=${PROJECT}

NAME
projects/{PROJECT}/logs/cloudaudit.googleapis.com%2Factivity
projects/{PROJECT}/logs/cloudaudit.googleapis.com%2Fdata_access

And can, e.g.:

FILTER="
log_id(\"cloudaudit.googleapis.com%2Fdata_access\") 
severity=ERROR
"
gcloud logging read "${FILTER}" \
--project=${PROJECT}

insertId: ...
logName: projects/{PROJECT}/logs/cloudaudit.googleapis.com%2Fdata_access
protoPayload:
  '@type': type.googleapis.com/google.cloud.audit.AuditLog
  authenticationInfo: {}
  authorizationInfo:
  - permission: secretmanager.secrets.get
    permissionType: ADMIN_READ
    resourceAttributes:
      name: projects/{NUMBER}/secrets/{NAME}
      service: secretmanager.googleapis.com
      type: secretmanager.googleapis.com/Secret
  methodName: google.cloud.secrets.v1beta1.SecretManagerService.GetSecret
  request:
    '@type': type.googleapis.com/google.cloud.secrets.v1beta1.GetSecretRequest
    name: projects/{PROJECT}/secrets/{NAME}
  requestMetadata:
    callerIp: ...
    callerSuppliedUserAgent: ...
    destinationAttributes: {}
    requestAttributes:
      auth: {}
      time: '2024-08-09T00:00:00.00000000Z'
  resourceName: projects/{NUMBER}/secrets/{NAME}
  serviceName: secretmanager.googleapis.com
  status:
    code: 7
    message: PERMISSION_DENIED
receiveTimestamp: '2024-08-09T00:00:00.000000000Z'
resource:
  labels:
    method: google.cloud.secrets.v1beta1.SecretManagerService.GetSecret
    project_id: {PROJECT}
    service: secretmanager.googleapis.com
  type: audited_resource
severity: ERROR
timestamp: '2024-08-09T00:00:00.000000000Z'

I don't understand why authenticationInfo is empty.

This may (!?) be a consequence of Data Access logging.