Azure devops pipeline: how to dynamically match a variable defined under variable group

I have a bash script in my ADO yaml pipeline that unzips a given set of files, some of these files are password protected. I have created a variable group and referenced it in my pipeline & defined the name of each password protected file as a key and their password as a secret value.

I would like to loop of over the target files and and for each file if there is a matching key or if its value is not empty then get the password defined under variable group. I have tried a few things but unfortunately no luck yet.

Variable group:

Key         Value
test.zip    ****  ->(testValue)

Pipeline:

- task: Bash@3
  inputs:
    targetType: 'inline'
    script: |
      file_dir="some_dir"
    
    
      if [ -d "$file_dir" ]; then
        echo "Listing files in $file_dir:"
      
        for file in "$file_dir"/*; do
          if [ -f "$file" ]; then
            build_var="$""("$file")"
            echo $build_var -> # prints $(test.zip)
            # Goal
            unzip -P "$(build_var)" test.zip -d .
            
          fi
        done
      fi

Since the pipeline processes lots of different files everyday I'm trying to find a way to not have to hardcode things as much as possible.

Is this possible?

Based on your description, you need to get the secret variable value in variable group based on the file name variable.

The secret variable will not be set in the Pipeline environment variable, so when you use the nested variable("$""("$file")") in bash script, it will not read the correct secret variable.

Since you doesn't want to hardcode the files in Pipeline, I afraid that you are not able to achieve the requirement in the single pipeline.

To meet your requirement, I suggest that you can split the pipelines into two. One is used to collect the file names(save to array) and trigger the other pipeline with the required array. The other pipeline is used to loop the file names and get the related secret password in variable group.

Here is an example:

variable Group:

file1.zip  ***
file2.zip  ***

Pipeline One:

pool:
  vmImage: ubuntu-latest

steps:
- task: Bash@3
  inputs:
    targetType: 'inline'
    script: |
      file_dir="file_pat"
      file_array=()
    
      if [ -d "$file_dir" ]; then
        echo "Listing files in $file_dir:"
      
        for file in "$file_dir"/*; do
           file_array+=$(basename $file,)
        done
      fi      
      echo $file_array

      az pipelines run --id {PipelintwoID} --organization $(System.CollectionUri) --project {YourProjectname} --parameters filenames="[$file_array]"
  env:
    AZURE_DEVOPS_EXT_PAT: $(PAT)

Pipeline Two: We can use format: $(${{filename}}) to get the secret variable in Variable Group

parameters:
- name: filenames
  type: object
  default: []
variables:
- group: filePass

steps:
- ${{ each filename in parameters.filenames }}:
  - bash: |
      echo ${{filename}}
    
      if [ -z "$(${{filename}})" ]
      then
        #variablevalue is null
            unzip  test.zip -d . 
      else
        #variablevalue is not null
            unzip -P "$(${{filename}})" test.zip -d .
      fi

Result: It will loop all file names in the array and run related commands in different tasks.

Here is the doc related to Azure DevOps CLI trigger Pipeline: az pipelines run