Google Cloud credentials with Terraform

I am using GCP provisioning using Terraform / Terragrunt, and I find the workflow with obtaining GCP credentials quite confusing. I've come from using AWS exclusively, where obtaining credentials, and configuring them in the AWS CLI was quite straightforward.

The Google Cloud Provider documentation states that you should define a provider block like so:

provider "google" {
  credentials = "${file("account.json")}"
  project     = "my-project-id"
  region      = "us-central1"
  zone        = "us-central1-c"
}

This credentials field shows I (apparently) must generate a service account, and keep a JSON somewhere on my filesystem.

However, if I run the command gcloud auth application-default login, this generates a token located at ~/.config/gcloud/application_default_credentials.json; alternatively I can also use gcloud auth login <my-username>. From there, I can access the Google API (which is what Terraform is doing under the hood as well) from the command line using a gcloud command.

So why does the Terraform provider require a JSON file of a service account? Why can't it just use the credentials that the gcloud CLI tool is already using?

If I configure Terraform to point to the application_default_credentials.json file, I get the following errors:

Initializing modules...

Initializing the backend...

Error: Failed to get existing workspaces: querying Cloud Storage failed: Get https://www.googleapis.com/storage/v1/b/terraform-state-bucket/o?alt=json&delimiter=%2F&pageToken=&prefix=projects%2Fsomeproject%2F&prettyPrint=false&projection=full&versions=false: private key should be a PEM or plain PKCS1 or PKCS8; parse error: asn1: syntax error: sequence truncated

if I configure Terraform to point to the application_default_credentials.json file, I get the following errors:

The credentials field in provider config expects a path to service account key file, not user account credentials file. If you want to authenticate with your user account try omitting credentials and then running gcloud auth application-default login; if Terraform doesn't find your credentials file you can set the GOOGLE_APPLICATION_CREDENTIALS environment variabe to point to ~/.config/gcloud/application_default_credentials.json.

Read here for more on the topic of service accounts vs user accounts. For what it's worth, Terraform docs explicitly advice against using application-default login:

This approach isn't recommended- some APIs are not compatible with credentials obtained through gcloud

Similarly GCP docs state the following:

Important: For almost all cases, whether you are developing locally or in a production application, you should use service accounts, rather than user accounts or API keys.