Azure Pipelines - Handling same variable names in different variable groups

I'm new to Azure Pipelines and I'm trying to handle the same variable name in different variable groups. I can't be too specific for security reasons, but the general gist is that I have multiple environments that I access via APIs, and I would like to be able transfer data any which way between these environments. The general set up is as such:

I have variable groups of different names (there are currently 2 for this poc, but will be 10-12 in prod)

'alpha'
'beta'

that contain the same variables:

'env' # hoping to use as a key from parameters to fetch this data
'url' # api url
'token' # api key

I would like to be able to get a source and destination environment from the user, and then set variables within the pipe sourceUrl, sourceEnv and sourceToken - destUrl, destEnv, destToken from these variable groups.

I'm aware on the documentation the variables with the same name are just set to the value of the last listed variable group, so I'm wondering if I have to use a bash script to set these variables dynamically. Hopefully the yaml makes this a little clearer:

trigger: none

parameters:
  - name: sourceEnv
    type: string
    default: alphaDev
    values:
      - alphaDev
      - betaDev
    displayName: 'Source Environment'

  - name: destEnv
    type: string
    default: betaDev
    values:
      - alphaDev
      - betaDev
    displayName: 'Destination Environment'

variables:
  - group: ${{ parameters.sourceEnv }} # i want to set sourceUrl sourceEnv and sourceToken to these values

  - group: ${{ parameters.destEnv }} # i want to set desturl destEnv and destToken to these values

# so essentially the variables would be:
  - name: sourceUrl
    value: ${{ parameters.sourceEnv}}.$(url)

  - name: sourceToken
    value: ${{ parameters.sourceEnv}}.$(token)

# and same for destination 

I have seen that you can directly declare the variable groups inside of jobs, which is what I'm doing currently with the POC.

However as the same variables are required across multiple jobs I wondered if there's a way to declare them once at the start of the file. Granted, this is mainly for maintainability, as once this is completed it will be others who manage the code.

As for the code, I wondered if setting variables immediately after calling the group might work

variables:
  - group: ${{ parameters.sourceEnv }} 
  - name: sourceUrl
    value: $(url)

  - group: ${{ parameters.destEnv }} 
  - name: destUrl
    value: $(url)

but this just sets them both to the destEnv url value.

Hopefully my requirements are clear, and perhaps this question is incredibly novel, but as I said I'm new to azure pipelines and am hoping someone is able to assist with with say a bash script to make this possible, unless there's a straight forward way I've missed :)

Based on your description, you have multiple variables with the same name in different variable groups.

Refer to this doc: Manage variable groups

If multiple variable groups include the same variable, the last variable group that uses the variable in the file sets the variable's value.

In this case, if you specify two variable groups directly in the root YAML node(based on your example in the question), it will use the value in the destEnv variable group. This is an expected behavior.

However as the same variables are required across multiple jobs I wondered if there's a way to declare them once at the start of the file.

For the reasons stated above, I'm afraid there is not such method can directly achieve this requirement.

so I'm wondering if I have to use a bash script to set these variables dynamically.

Yes. This is required. You need to set the variables group in two different jobs/stages and use cross jobs/stages variables to map the correct value to source and dest env varaibles. Refer to this doc: Use output variables from tasks

Here is an example:

parameters:
  - name: sourceEnv
    type: string
    default: alphaDev
    values:
      - alphaDev
      - betaDev
    displayName: 'Source Environment'

  - name: destEnv
    type: string
    default: betaDev
    values:
      - alphaDev
      - betaDev
    displayName: 'Destination Environment'

jobs:
- job: SourceEnv
  dependsOn:
  variables:
    - group: ${{ parameters.sourceEnv }} 
  steps:
    - bash: |
        echo "##vso[task.setvariable variable=sourceUrl;isOutput=true]${{ parameters.sourceEnv}}.$(url)"
        echo "##vso[task.setvariable variable=sourceToken;isOutput=true]${{ parameters.sourceEnv}}.$(token)"
      name: setvarStep

- job: DestEnv
  dependsOn:
  variables:
    - group: ${{ parameters.destEnv }}
  steps:
    - bash: |
        echo "##vso[task.setvariable variable=destUrl;isOutput=true]${{ parameters.destEnv}}.$(url)"
        echo "##vso[task.setvariable variable=destToken;isOutput=true]${{ parameters.destEnv}}.$(token)"
      name: setvarStep

- job: UseVariables
  dependsOn:
    - SourceEnv
    - DestEnv
  variables:
    sourceUrl: "$[ dependencies.SourceEnv.outputs['setvarStep.sourceUrl'] ]"
    sourceToken: "$[ dependencies.SourceEnv.outputs['setvarStep.sourceToken'] ]"
    destUrl: "$[ dependencies.DestEnv.outputs['setvarStep.destUrl'] ]"
    destToken: "$[ dependencies.DestEnv.outputs['setvarStep.destToken'] ]"
  steps:
    - script: |
        echo $(sourceUrl)
        echo $(sourceToken)
        echo $(destUrl)
        echo $(destToken)

On the other hand, whether we use classic mode or YAML, it is not recommended to define a variable with the same name in different variable groups. You can also consider setting the variable name based on the environment name in variable group.