azureazure-devopsazure-pipelinesazure-keyvault

Maintaining secrets in Azure Key Vault so that we can limit individual access


I’m trying to understand the best practices around maintaining secrets in key vaults:

I was trying to implement the second option so that we can have all the secrets managed from a single place and this is what happens.

The plan was to have those variables that are supposed to be in KeyVault start with the static string KeyVault_ and then read those using the az pipeline variable group show -- group-id $GROUP_ID but as expected the value is empty for secrets values.

How can I achieve a similar approach and update the key vault secrets from ADO secrets.

$variable_group.variables.GetEnumerator() | ForEach-Object { 
    if ($_.Key.StartsWith("KeyVault_")) {
        $SecretName = $_.Key -replace "KeyVault_", ""
        $SecretValue = $_.Value.value

        if ($SecretValue) {
            Write-Output "Creating secret for $SecretName."
            az keyvault secret set --vault-name $KeyVaultName --name $SecretName --value $SecretValue --output none
            Write-Output "Added secret '$SecretName' to Key Vault '$KeyVaultName'."
        } else {
            Write-Warning "No value found for variable '$($_.Key)'."
        }
    }
}

How can I get the value for $SecretValue assigned properly, I've tried $($_.Key) but with no luck!


Solution

  • In the Azure DevOps project where the variable group is in, you can set up 2 YAML pipelines (ListVars and UpdateSecrets) like as below:

    enter image description here

    1. The PowerShell script "scripts/list-variabe-names.ps1".

      param (
          # Name of Azure DevOps organization.
          [Alias("org", "o")]
          [string] $organization,
      
          # Name of project where the variable group and pipelines are in.
          [Alias("proj", "p")]
          [string] $project,
      
          # ID of variable group.
          [Alias("gid")]
          [string] $groupId,
      
          # ID of the pipeline (UpdateSecrets) that needs to be triggered by this script.
          [Alias("pid")]
          [string] $PipelineId,
      
          # Name of Azure Key Vault.
          [Alias("kv")]
          [string] $keyVault
      )
      
      $response = (az pipelines variable-group variable list --org "https://dev.azure.com/$organization" -p "$project" --id $groupId | ConvertFrom-Json)
      $varNames = $response | Get-Member | Where {$_. MemberType -like 'NoteProperty'} | Select-Object -Property name
      
      $varNamesList = "["
      ForEach ($varName in $varNames.Name)
      {
          $varNamesList += "${varName},"
      }
      
      $varNamesList = $varNamesList -replace ".{1}$"
      $varNamesList += "]"
      
      az pipelines run --org "https://dev.azure.com/$organization" -p "$project" --id $PipelineId --parameters variableNames="$varNamesList" KeyVaultName="$keyVault"
      
    2. The first pipeline (ListVars).

      # list-variable-group-variables.yml
      
      trigger: none
      
      # When manually triggering this pipeline,
      # you can enter new values to override the default values of parameters.
      parameters:
      - name: organizationName
        type: string
        default: 'myOrg'
      
      - name: projectName
        type: string
        default: 'myProj'
      
      - name: groupId
        type: number
        default: 5
      
      - name: PipelineId
        type: number
        default: 83
      
      - name: KeyVaultName
        type: string
        default: 'myKV'
      
      steps:
      - task: PowerShell@2
        env:
          AZURE_DEVOPS_EXT_PAT: $(System.AccessToken)
        inputs:
          pwsh: true
          filePath: 'scripts/list-variabe-names.ps1'
          arguments: >
            -o "${{ parameters.organizationName }}"
            -p "${{ parameters.projectName }}"
            -gid ${{ parameters.groupId }}
            -pid ${{ parameters.PipelineId }}
            -kv "${{ parameters.KeyVaultName }}"
      

      enter image description here

    3. The second pipeline (UpdateSecrets).

      # update-secrets.yml
      
      trigger: none
      
      parameters:
      - name: variableNames
        type: object
        default: []
      
      - name: KeyVaultName
        type: string
        default: ''
      
      variables:
      - group: myVarGroup
      
      steps:
      - ${{ each variableName in parameters.variableNames }}:  
        - task: AzureCLI@2
          displayName: 'Set KeyVault secret - ${{ variableName }}'
          inputs:
            azureSubscription: 'myArmConnection'
            scriptType: 'pscore'
            scriptLocation: 'inlineScript'
            inlineScript: |
              Write-Host "Setting Azure Key Vault secret - ${{ variableName }}"
              az keyvault secret set --vault-name ${{ parameters.KeyVaultName }} --name ${{ variableName }} --value $(${{ variableName }})
      
    4. On the Security hub of Pipelines, ensure the build identities "Project Collection Build Service ({Organization-Name})" and "{Project-Name} Build Service ({Organization-Name})" have the following permissions set to "Allow".

      • Queue builds
      • Edit queue build configuration

      enter image description here

    With above configurations, when you manually trigger the first pipeline (ListVars) with your values of parameters, the pipeline will execute the PowerShell script "scripts/list-variabe-names.ps1" to do the following things.

    When the second pipeline is triggered, it receives the parameter values passed from the first pipeline, and then use the values to run the Azure CLI "az keyvault secret set" to set or update secret with the new secret value into Azure Key Vault.