Gitlab connection to GCP Workload Identity returning invalid_grant on a JWKs parsing error

I have set up a Workload Identity Federation with an on-premise gitlab installation, the provider status shows as "Enabled" and healthy in the console.

The part I don't get past is specifically this:

  - |
    PAYLOAD=$(cat <<EOF
    {
    "audience": "//iam.googleapis.com/projects/${PROJECT_NUMBER}/locations/global/workloadIdentityPools/${POOL_ID}/providers/${PROVIDER_ID}",
    "grantType": "urn:ietf:params:oauth:grant-type:token-exchange",
    "requestedTokenType": "urn:ietf:params:oauth:token-type:access_token",
    "scope": "https://www.googleapis.com/auth/cloud-platform",
    "subjectTokenType": "urn:ietf:params:oauth:token-type:jwt",
    "subjectToken": "${CI_JOB_JWT_V2}"
    }
    EOF
    )
  - echo $PAYLOAD
  - |
    FEDERATED_TOKEN=$(curl -X POST "https://sts.googleapis.com/v1/token"
    --header "Accept: application/json"
    --header "Content-Type: application/json"
    --data "${PAYLOAD}"
    )
  - echo $FEDERATED_TOKEN

Compared to the Gitlab Working Example this seems correct enough, if I change the audience I get errors on that my audience value is incorrect so it seems to at least talk to the server.

This is the error response for FEDERATED_TOKEN:

{"error":"invalid_grant","error_description":"Parsing error for JWKs: [Line 0, column 0: Unexpected end of stream : expected '{']"}

I've tried googling around for the Parsing error for JWKs but haven't learned anything useful, does anyone have any guidance?

Thanks in advance!

Update after comments from before June 20th

Adding or removing --header "Accept: application/json" did not do much. The subjectToken seems to be correct:

Header part:

{"alg":"RS256","kid":"[REDACTED]","typ":"JWT"}

Payload part:

{"namespace_id":"46","namespace_path":"devops","project_id":"682","project_path":"devops/terraform-source","user_id":"REDACTED","user_login":"chase","user_email":"REDACTED","pipeline_id":"124241","pipeline_source":"push","job_id":"1131468","ref":"workload-identity-debugging","ref_type":"branch","ref_protected":"false","jti":"REDACTED-1d58-43f6-9b92-72914b937023","iss":"REDACTED","iat":REDACTED,"nbf":REDACTED,"exp":REDACTED,"sub":"project_path:devops/terraform-source:ref_type:branch:ref:workload-identity-debugging","aud":"REDACTED"}

The signature is hard to decrypt/debug.

The header, payload and signature are formatted in base64 and as expted concatted together using . (dots).

gcloud

Another interesting behavior I noticed when trying to grab an access token using gcloud directly:

$ echo ${CI_JOB_JWT_V2} > .ci_job_jwt_file
$ gcloud iam workload-identity-pools create-cred-config "projects/REDACTED/locations/global/workloadIdentityPools/${POOL_ID}/providers/${PROVIDER_ID}" --service-account="${SERVICE_ACCOUNT_EMAIL}" --output-file=.gcp_temp_cred.json --credential-source-file=.ci_job_jwt_file
Created credential configuration file [.gcp_temp_cred.json].
$ gcloud auth login --cred-file=`pwd`/.gcp_temp_cred.json
Authenticated with external account credentials for: [terraformer@REDACTED.iam.gserviceaccount.com].
Your current project is [None].  You can change this setting by running:
  $ gcloud config set project PROJECT_ID
$ gcloud auth list
                    Credentialed Accounts
ACTIVE  ACCOUNT
*       terraformer@REDACTED.iam.gserviceaccount.com
To set the active account, run:
    $ gcloud config set account `ACCOUNT`
$ echo "GOOGLE_APPLICATION_CREDENTIALS=$(gcloud auth print-access-token)"
ERROR: (gcloud.auth.print-access-token) ("Error code invalid_grant: Parsing error for JWKs: [Line 0, column 0: Unexpected end of stream : expected '{']", '{"error":"invalid_grant","error_description":"Parsing error for JWKs: [Line 0, column 0: Unexpected end of stream : expected \'{\']"}')
GOOGLE_APPLICATION_CREDENTIALS=

Which seems to be the same thing I'm looking at from sts directly.

Cheers, Chase

Turns out these are symptoms of Google Workload Identity Federation not being able to 1) connect to your issuer (in my case, Gitlab) 2) parse the jwks_uri.

From Verification of external credentials we can learn two things:

• there's a connection being made to $ISSUER/.well-known/openid-configuration AKA the discovery document (so make sure that is reachable)
• there's a follow-up request being made to the value of jwks_uri in the discovery document (make sure that is also reachable)

Finally, if your jwks_uri is returning a 500 error, it is discussed here that you should either add double quotes around the openid_connect_signing_key (found in rails-secret) or replace the \n to multiline. After restarting my gitlab deployment, I was able to hit the jwks_uri (and so was Workload Identity) - after which I was able to generate access tokens.

Problem solved!