Invalid Terraform AWS provider credentials when passing AWS system manager parameter store variables

Background:

I'm using an AWS CodeBuild buildspec.yml to iterate through directories from a GitHub repo to apply IaC using Terraform. To access the credentials needed for the Terraform AWS provider, I used AWS system manager parameter store to retrieve the access and secret key within the buildspec.yml.

Problem:

The system manager parameter store masks the access and secret key env value so when they are inherited by the Terraform AWS provider, the provider outputs that the credentials are invalid:

Error: error configuring Terraform AWS Provider: error validating provider credentials: error calling sts:GetCallerIdentity: InvalidClientTokenId: The security token included in the request is invalid.
    status code: 403, request id: xxxx

To reproduce the problem:

• Create system manager parameter store variables (TF_VAR_AWS_ACCESS_KEY_ID=access, TF_AWS_SECRET_ACCESS_KEY=secret)

• Create AWS CodeBuild project with:

 "source": {
                "type": "NO_SOURCE",
}
"environment": {
                "type": "LINUX_CONTAINER",
                "image": "aws/codebuild/standard:4.0",
                "computeType": "BUILD_GENERAL1_SMALL"
}

buildspec.yml with the following: (modified to create .tf files instead of sourcing from github)

version: 0.2
env:
  shell: bash
  parameter-store:
    TF_VAR_AWS_ACCESS_KEY_ID: TF_AWS_ACCESS_KEY_ID
    TF_VAR_AWS_SECRET_ACCESS_KEY: TF_AWS_SECRET_ACCESS_KEY
phases:
  install:
    commands:
      - wget https://releases.hashicorp.com/terraform/0.12.28/terraform_0.12.28_linux_amd64.zip -q
      - unzip terraform_0.12.28_linux_amd64.zip && mv terraform /usr/local/bin/
      - printf "provider "aws" {\n\taccess_key = var.AWS_ACCESS_KEY_ID\n\tsecret_key = var.AWS_SECRET_ACCESS_KEY\n\tversion    = \"~> 3.2.0\"\n}" >> provider.tf
      - printf "variable "AWS_ACCESS_KEY_ID" {}\nvariable "AWS_SECRET_ACCESS_KEY" {}" > vars.tf
      - printf "resource \"aws_s3_bucket\" \"test\" {\n\tbucket = \"test\"\n\tacl = \"private\"\n}" >> s3.tf
      - terraform init
      - terraform plan

Attempts:

1. Passing creds through terraform -vars option:

terraform plan -var="AWS_ACCESS_KEY_ID=$TF_VAR_AWS_ACCESS_KEY_ID" -var="AWS_ACCESS_KEY_ID=$TF_VAR_AWS_SECRET_ACCESS_KEY"

but I get the same invalid credentials error

2. Export system manager parameter store credentials within buildspec.yml:

commands:
  - export AWS_ACCESS_KEY_ID=$TF_VAR_AWS_ACCESS_KEY_ID
  - export AWS_SECRET_ACCESS_KEY=$TF_VAR_AWS_SECRET_ACCESS_KEY

which results in duplicate masked variables and the same error above. printenv output within buildspec.yml:

AWS_ACCESS_KEY_ID=***
TF_VAR_AWS_ACCESS_KEY_ID=***
AWS_SECRET_ACCESS_KEY=***
TF_VAR_AWS_SECRET_ACCESS_KEY=***

Possible solution routes:

• Somehow pass the MASKED parameter store credential values into Terraform successfully (preferred)
• Pass sensitive credentials into the Terraform AWS provider using a different method e.g. AWS secret manager, IAM role, etc.
• Unmask the parameter store variables to pass into the aws provider (probably defeats the purpose of using aws system manager in the first place)

Pass sensitive credentials into the Terraform AWS provider using a different method e.g. AWS secret manager, IAM role, etc.

Generally you wouldn't need to hard-code AWS credentials for terraform to work. Instead CodeBuild IAM role should be enough for terraform, as explain in terraform docs.

Having this in mind, I verified that the following works and creates the bucket requested using terraform from CodeBuild project. The default CB role was modified with S3 permissions to allow creation of the bucket.

version: 0.2
phases:
  install:
    commands:
      - wget https://releases.hashicorp.com/terraform/0.12.28/terraform_0.12.28_linux_amd64.zip -q
      - unzip terraform_0.12.28_linux_amd64.zip && mv terraform /usr/local/bin/
      - printf "resource \"aws_s3_bucket\" \"test\" {\n\tbucket = \"test-43242-efdfdfd-4444334\"\n\tacl = \"private\"\n}" >> s3.tf
      - terraform init
      - terraform plan
      - terraform apply -auto-approve