In the code below, I have a variable (storage_account_name) being output in the first job, within the task called TaskName. I want to be able to retrieve its value in a second job (deployment job), but the variable is empty.

jobs:
- job: dependency
  steps:
  - template: my_template.yml

- deployment: my_deployment
  environment: MyEnvironment
  dependsOn: 
  - dependency
  variables:
  - name: storage_account_name
    value: $[ dependencies.dependency.outputs['TaskName.storage_account_name'] ]
  strategy:
    runOnce:
      deploy:
        steps:
        - checkout: self
        - template: another_template.yml
          parameters: 
            TF_VAR_storage_account_name: $(storage_account_name)

The exact same code works well when the second job is a normal job. Any idea what I am doing wrong?

Within my_template.yml I am running a Bash task that calls a script:

- task: Bash@3
  name: TaskName
  inputs:
    targetType: filePath
    filePath: scripts/my_script.sh
    failOnStderr: true

on my_script.sh I then have:

if [ "$IS_SENSITIVE" == "true" ]; then
  echo seeding the sensitive "$KEY" output variable...
  echo "##vso[task.setvariable variable=$KEY;isOutput=true;issecret=true;]$VALUE"
else
  echo seeding the "$KEY" output variable...
  echo "##vso[task.setvariable variable=$KEY;isOutput=true;]$VALUE"
fi

The {Key -> Value} pairs are outputs of running terraform commands. This exact script is working fine in other projects

As per Deployment jobs - Support for output variables, you need to use a different syntax for variables and conditions in deployment jobs.

Also, each deployment strategy (runOnce, canary, rolling) uses a different syntax.

Example

The following pipeline shows how to define and use variables created in both a standard and deployment jobs:

trigger: none

pool:
  vmImage: 'ubuntu-latest'

jobs:
  - job: jobA
    displayName: 'Set variable A in job'
    steps:
      - checkout: none
      - script: |
          echo "##vso[task.setvariable variable=A;isOutput=true]AAAA"
        displayName: 'Set variable A'
        name: setVariableATask
  
  - deployment: deploymentJobB
    displayName: 'Set variable B in deployment job'
    environment: 'no-approvals'
    strategy:
      runOnce:
        deploy:
          steps:
            - script: |
                echo "##vso[task.setvariable variable=B;isOutput=true]BBBB"
              displayName: 'Set variable A'
              name: setVariableBTask

  - job: printVariablesJob
    displayName: 'Print variables in job'
    dependsOn: 
      - jobA
      - deploymentJobB
    variables:
      varFromJob: $[ dependencies.jobA.outputs['setVariableATask.A'] ]
      varFromDeploymentJob: $[ dependencies.deploymentJobB.outputs['deploymentJobB.setVariableBTask.B'] ]
    steps:
      - checkout: none
      - script: |
          echo "varFromJob: $(varFromJob)"
          echo "varFromDeploymentJob: $(varFromDeploymentJob)"
        displayName: 'Print variables'
  
  - deployment: printVariablesDeploymentJob
    displayName: 'Print variables in deployment job'
    dependsOn: 
      - jobA
      - deploymentJobB
    variables:
      varFromJob: $[ dependencies.jobA.outputs['setVariableATask.A'] ]
      varFromDeploymentJob: $[ dependencies.deploymentJobB.outputs['deploymentJobB.setVariableBTask.B'] ] 
    environment: 'no-approvals'
    strategy:
      runOnce:
        deploy:
          steps:
            - script: |
                echo "varFromJob: $(varFromJob)"
                echo "varFromDeploymentJob: $(varFromDeploymentJob)"
              displayName: 'Print variables'

The exact same code works well when the second job is a normal job.

I don't have information about how the variable was set. I was not able to reproduce this issue using the above pipeline.

Using the same syntax works whether I use a standard or deployment job to reference the variables:

variables:
  # $[ dependencies.{job-name}.outputs['{task-name}.{variable-name}'] ]
  varFromJob: $[ dependencies.jobA.outputs['setVariableATask.A'] ]

  # $[ dependencies.{deployment-job-name}.outputs['{deployment-job-name}.{task-name}.{variable-name}'] ]
  varFromDeploymentJob: $[ dependencies.deploymentJobB.outputs['deploymentJobB.setVariableBTask.B'] ]