Terraform Cloud not recognizing TF_VAR environment variable in Github Actions

In my Github actions pipeline I use the Terraform Cloud workflows like this:

terraform:
    name: "Terraform Plan"
    runs-on: ubuntu-latest
    needs: build
    permissions:
      contents: read
      pull-requests: write
    env:
      TF_CLOUD_ORGANIZATION: "xxx"
      TF_API_TOKEN: "${{ secrets.TF_API_TOKEN }}"
      TF_WORKSPACE: "xxx"
      CONFIG_DIRECTORY: "./infrastructure"
    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Set variables
        run: echo "TF_VAR_image_tag=$(echo $GITHUB_SHA | cut -c 1-6)" >> $GITHUB_ENV

      - name: Upload Configuration
        uses: hashicorp/tfc-workflows-github/actions/upload-configuration@v1.0.4
        id: plan-upload
        with:
          directory: ./
          workspace: ${{ env.TF_WORKSPACE }}
          speculative: true

      - name: Create Plan Run
        uses: hashicorp/tfc-workflows-github/actions/create-run@v1.0.4
        id: plan-run
        with:
          workspace: ${{ env.TF_WORKSPACE }}
          configuration_version: ${{ steps.plan-upload.outputs.configuration_version_id }}
          plan_only: true

In which I try to set image_tag Terraform variable by setting the TF_VAR_image_tag environment variable. However, I keep getting The root module input variable \"image_tag\" is not set, and has no default value. Which implies that Terraform did not receive a value for that variable. I defined the image_tag like so:

variable "image_tag" {
  type = string
}

What am I missing?

You are using the "create-run" GitHub Action which internally runs the "tfci" tool, and is specifically running the tfci run create command.

tfci is an alternative to Terraform CLI which more directly follows the shape of the Terraform Cloud API, whereas Terraform CLI's cloud integration is designed to adapt the normal Terraform CLI workflow to work with Terraform Cloud.

That tool implements a similar TF_VAR_ convention as Terraform CLI does, but it has a significant difference: it doesn't have access to the configuration and so it cannot detect automatically what types each of your input variables has. Terraform CLI uses the type of the variable to make it more convenient to pass the value for a string variable without having to write it in full Terraform expression syntax (with quotes), but tfci requires all input variables to be written in the full Terraform expression syntax.

In your case, that means that your TF_VAR_image_tag environment variable value must contain a valid HCL string literal, including the quotes, like "abc123", instead of just a naked token like abc123.

Unfortunately I cannot find any documentation on exactly what format is expected for the file that GITHUB_ENV refers to -- the documentation only describes how to append to it with some trivial examples, and doesn't document exactly what syntax GitHub is expecting -- but assuming that GitHub parses this format in a way that treats quotes literally then the following might work:

echo "TF_VAR_image_tag=\"$(echo $GITHUB_SHA | cut -c 1-6)\"" >> $GITHUB_ENV

After being interpreted by the shell, the GITHUB_ENV file would contain something like the following:

TF_VAR_image_tag="abc123"

...whereas your current code would generate this:

TF_VAR_image_tag=abc123

Internally within Terraform Cloud, the execution environment will generate a terraform.tfvars file that includes image_tag = followed by literally whatever you put in that environment variable. So without the quotes, the syntax of that file would be invalid:

image_tag = abc123

That's invalid because abc123 is the syntax for a reference to a symbol/variable, and those are not allowed in .tfvars files. Terraform will report "variables are not allowed here".

Instead, the generated file must contain this, with the quotes around the string:

image_tag = "abc123"

...and so the environment variable value MUST also include those quotes, so that the generated terraform.tfvars file will be valid.