GCP authentication fails in GitHub actions - IaC with Terraform

I am facing the problem described below. My GitHub actions fail when I am trying to authenticate to Google Cloud in my CICD pipeline.

My common-terraform.yml -> 1st tested version:

name: 'Common Terraform Workflow'

on:
  workflow_call:

jobs:
  terraform:
    name: 'Terraform'
    runs-on: ubuntu-latest

    # Use the Bash shell regardless whether the GitHub Actions runner is ubuntu-latest, macos-latest, or windows-latest
    defaults:
      run:
        shell: bash
        working-directory: ./terraform

    steps:
    # Checkout the repository to the GitHub Actions runner
    - name: Checkout
      uses: actions/checkout@v4

    # Install the latest version of Terraform CLI
    - name: Setup Terraform
      uses: hashicorp/setup-terraform@v1

    - name: Setup terraform variables
      id: vars
      run: |-
        cat > pipeline.auto.tfvars <<EOF
        project_id="${{ vars.PROJECT_ID }}" 
        EOF    

    # Initialize a new or existing Terraform working directory by creating initial files, loading any remote state, downloading modules, etc.
    - name: Terraform Init
      run: terraform init
      env:
        GOOGLE_CREDENTIALS: ${{ secrets.TF_GOOGLE_CREDENTIALS }}

    # Run terraform fmt to check whether the formatting of the files is correct
    - name: Terraform Format
      run: terraform fmt -check
      env:
        TF_VAR_schema_path: $GITHUB_WORKSPACE/terraform/schemas

    # Run terraform plan
    - name: Terraform Plan
      run: terraform plan
      env:
        TF_VAR_schema_path: $GITHUB_WORKSPACE/terraform/schemas

    # Run terraform apply
    - name: Terraform Apply
      run: terraform apply -auto-approve
      env:
        TF_VAR_schema_path: $GITHUB_WORKSPACE/terraform/schemas

terraform-dev.yml -> the same in both examples:

name: 'Terraform Dev'

on:
  push:
    branches:
      - '**'
      - '!main'
  pull_request:
    branches:
      - '**'
      - '!main'

jobs:
  call-common-terraform:
    uses: ./.github/workflows/common-terraform.yml

1st example error:

Run terraform init
  terraform init
  shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
  env:
    TERRAFORM_CLI_PATH: /home/runner/work/_temp/7325049d-c6a7-480a-9c4f-2c34d758baec
    GOOGLE_CREDENTIALS: 
/home/runner/work/_temp/7325049d-c6a7-480a-9c4f-2c34d758baec/terraform-bin init
Initializing the backend...
Initializing modules...
- pubsub-bq in modules/020-pubsub
Downloading registry.terraform.io/terraform-google-modules/pubsub/google 7.0.0 for pubsub-bq.pubsub-bq...
- pubsub-bq.pubsub-bq in .terraform/modules/pubsub-bq.pubsub-bq
- service-account in modules/010-sa
╷
│ Error: storage.NewClient() failed: dialing: google: could not find default credentials. See https://cloud.google.com/docs/authentication/external/set-up-adc for more information
│ 
│ 
╵

Warning: The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/

Warning: The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/

Warning: The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
Error: Terraform exited with code 1.
Error: Process completed with exit code 1.

My common-terraform.yml -> 2nd tested version:

name: 'Common Terraform Workflow'

on:
  workflow_call:

jobs:
  terraform:
    name: 'Terraform'
    runs-on: ubuntu-latest

    # Use the Bash shell regardless whether the GitHub Actions runner is ubuntu-latest, macos-latest, or windows-latest
    defaults:
      run:
        shell: bash
        working-directory: ./terraform

    steps:
    # Checkout the repository to the GitHub Actions runner
    - name: Checkout
      uses: actions/checkout@v4

    # Authenticate with the Google Cloud service account key
    - name: GCP Auth
      uses: 'google-github-actions/auth@v2'
      with:
        credentials_json: '${{ secrets.TF_GOOGLE_CREDENTIALS }}'

    # Install the latest version of Terraform CLI
    - name: Setup Terraform
      uses: hashicorp/setup-terraform@v1

    - name: Setup terraform variables
      id: vars
      run: |-
        cat > pipeline.auto.tfvars <<EOF
        project_id="${{ vars.PROJECT_ID }}" 
        EOF    

    # Initialize a new or existing Terraform working directory by creating initial files, loading any remote state, downloading modules, etc.
    - name: Terraform Init
      run: terraform init

    # Run terraform fmt to check whether the formatting of the files is correct
    - name: Terraform Format
      run: terraform fmt -check
      env:
        TF_VAR_schema_path: $GITHUB_WORKSPACE/terraform/schemas

    # Run terraform plan
    - name: Terraform Plan
      run: terraform plan
      env:
        TF_VAR_schema_path: $GITHUB_WORKSPACE/terraform/schemas

    # Run terraform apply
    - name: Terraform Apply
      run: terraform apply -auto-approve
      env:
        TF_VAR_schema_path: $GITHUB_WORKSPACE/terraform/schemas

2nd example error:

Run google-github-actions/auth@v2
  with:
    create_credentials_file: true
    export_environment_variables: true
    universe: googleapis.com
    cleanup_credentials: true
    access_token_lifetime: 3600s
    access_token_scopes: https://www.googleapis.com/auth/cloud-platform
    id_token_include_email: false
Error: google-github-actions/auth failed with: the GitHub Action workflow must specify exactly one of "workload_identity_provider" or "credentials_json"! If you are specifying input values via GitHub secrets, ensure the secret is being injected into the environment. By default, secrets are not passed to workflows triggered from forks, including Dependabot.

It looks like the credentials managed with GitHub secrets does not work properly.

TF_GOOGLE_CREDENTIALS provided to GitHub: Credentials were provided as 1 line without spaces and new lines characters ('\n').

{
  "type": "service_account",
  "project_id": "xyz",
  "private_key_id": "xyz",
  "private_key": "-----BEGIN PRIVATE KEY----- xyz -----END PRIVATE KEY----- ",
  "client_email": "test-sa@xyz.iam.gserviceaccount.com",
  "client_id": "123",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/test-sa%40xyz.iam.gserviceaccount.com",
  "universe_domain": "googleapis.com"
}

Do you have any suggestions what can be a reason for my problem?

You are using workflow_call, it means that there is a main Github action file (caller) that is calling this common file (called). In the second example, clearly credentials_json is not being passed to the called file.

What you need to do is adding secrets: inherit to the job in caller file:

job:
  uses: ./.github/workflows/common-terraform.yml
  secrets: inherit

By secrets: inherit, all the secrets are sent to common-terraform.yml. Or you can choose the secrets that you want to pass:

job:
  uses: ./.github/workflows/common-terraform.yml
  secrets:
    credentials_json: ${{ secrets.TF_GOOGLE_CREDENTIALS }}

Let's say that you have used secrets: inherit. Then in your common-terraform.yml file, use the inputs and secrets keywords to define inputs or secrets that will be passed from a caller workflow.

on:
  workflow_call:
    secrets:
      TF_GOOGLE_CREDENTIALS:
        required: true

The rest of common-terraform.yml would be the same