create Azure Keyvault secret without exposing values

My requirement is to automate the creation of secrets in Keyvault without exposing the values of these secrets during the pipeline execution(AzureDevops Server).

tried, different options such as having input paramaters (it will display the values when its perform initialization) , runtime variables etc (list of variables, or dynamic creation as per input is not possible), but couldn't fulfil our requirements.

Finally planned to use the below script to read values from library group and create the values in key vault. But when we are masking the secret values in Library group (by locking), the secrets are getting created with null value only. So is there any way to retrieve the values secret variables and create them in key vault only if the secret value is different(if existing)

-bash: | 
   az login --service-principal --username $(spid) --password $(spsecret) --tenant $(tenantid)
   az account set --subscription ${{ variables.subscription }}
   az config set extension.use_dynamic_install=yes_without_prompt
   groupID=`az pipelines variable-group list -p myproject --group-name ${{ environment }}-${{ parameters.myapp}}-kv-secret --query '[].id' -o tsv`
   echo "grouup id is $groupID"
   variables=$(az pipelines variable-group variable list -p myproject --group-id $groupID --output json)
   echo "$variables" | jq -r 'keys[] as $k | "\($k)=\(.[$k].value)"' | while read variable; do
   name=$(echo "$variable" | cut -d= -f1)
   value=$(echo "$variable" | cut -d= -f2)
   if ! az keyvault secret show --vault-name "$(${{ variables.podkv }})" --name "$name" &>/dev/null; then
     expiryDate=$(date -u -d '+2 years' '+%Y-%m-%dT%H:%MZ')
     az keyvault secret set --vault-name "$(${{ variables.podkv }})" --name "$name" --value "$value" --expires $expiryDate
     secretValue=$(az keyvault secret show --vault-name "$(${{ variables.podkv }})" --name "$name" --query value --output tsv)
     if [[ "$secretValue" != "$value" ]]; then
       echo "Failed to create secret $secretName in Key Vault $keyVaultName"
     exit 1
     fi
     if [[ "$secretValue" == "$value" ]]; then
       echo "Created secret $secretName in Key Vault $keyVaultName, hence cleaning the variable from the keyvault"
       az pipelines variable-group variable delete --group-id $groupID --name "$name --yes"
     fi
   fi
   done

---

Updated script,

where the secret is getting updated each time even though the same value is in the file.

while IFS= read -r line    
do    
  name=$(echo "$line" | cut -d '=' -f 1)    
  value=$(echo "$line" | cut -d '=' -f 2)    
  if ! az keyvault secret show --vault-name "$(${{ variables.podkv }})" --name "$name" &>/dev/null; then    
    expiryDate=$(date -u -d '+2 years' '+%Y-%m-%dT%H:%MZ')    
    az keyvault secret set --vault-name "$(${{ variables.podkv }})" --name "$name" --value "$value" --expires $expiryDate --output none  
    secretValue=$(az keyvault secret show --vault-name "$(${{ variables.podkv }})" --name "$name" --query value --output tsv)    
    if [[ "$secretValue" != "$value" ]]; then    
      echo "Failed to create secret $name in Key Vault ${{ variables.podkv }}"    
      exit 1    
    else    
      echo "Created secret $name in Key Vault ${{ variables.podkv }}, hence cleaning the variable from the keyvault"    
    fi    
  else    
    echo "The secretName $name already exists, so checking if the secret value is the same"    
    existing_secretvalue=$(az keyvault secret show --name "$name" --vault-name "$(${{ variables.podkv }})" --query "value")    
    if [[ "$existing_secretValue" != "$value" ]]; then    
      echo "there is secret value change for the secretname $name , so Creating a new secret value for the existing secret $name"    
      expiryDate=$(date -u -d '+2 years' '+%Y-%m-%dT%H:%MZ')    
      az keyvault secret set --vault-name "$(${{ variables.podkv }})" --name "$name" --value "$value" --expires $expiryDate --output none  
      new_secretValue=$(az keyvault secret show --vault-name "$(${{ variables.podkv }})" --name "$name" --query value --output tsv)    
      if [[ "$new_secretValue" != "$value" ]]; then    
        echo "Failed to create secret $name in Key Vault ${{ variables.podkv }}"    
        exit 1    
      else    
        echo "Created secret $name in Key Vault ${{ variables.podkv }}, hence cleaning the variable from the keyvault"    
      fi  
    fi  
  fi  
done < "$(keyvault.secureFilePath)"

when we are masking the secret values in Library group (by locking), the secrets are getting created with null value only.

This is an expected behavior. When we set the secret variable in Variable Group, we are not able to use Rest API or Azure DevOps CLI to get the variable. It will show as NULL Value.

In this case, I am afraid that the variable group method can not meet your requirement.

is there any way to retrieve the values secret variables

Based on your description, I suggest that you can save the variables in a file and add the file to Azure DevOps Secure file(Pipelines -> Library -> Secure file).

The secure file can only be downloaded with the Download secure file v1 task in Azure Pipelines.

Here are the steps:

Step1: We can create a file and save the variables.

For example:

test.txt

var1=value1
var2=value2
var3=value3

Step2: Upload the file to secure file.

For example:

Step3: We can use the Download secure file v1 task to download the secure file and then use the bash script to loop the values in the file. Finally we can set the Key vault value.

Here is an example:

steps:
- task: DownloadSecureFile@1
  name: keyvault
  inputs:
    secureFile: 'test.txt'
- bash: |
    while IFS= read -r line
    do
        name=$(echo "$line" | cut -d '=' -f 1)
        value=$(echo "$line" | cut -d '=' -f 2)

      if ! az keyvault secret show --vault-name "$(${{ variables.podkv }})" --name "$name" &>/dev/null; then
      expiryDate=$(date -u -d '+2 years' '+%Y-%m-%dT%H:%MZ')
      az keyvault secret set --vault-name "$(${{ variables.podkv }})" --name "$name" --value "$value" --expires $expiryDate
      secretValue=$(az keyvault secret show --vault-name "$(${{ variables.podkv }})" --name "$name" --query value --output tsv)
      if [[ "$secretValue" != "$value" ]]; then
        echo "Failed to create secret $secretName in Key Vault $keyVaultName"
      exit 1
      fi
      if [[ "$secretValue" == "$value" ]]; then
        echo "Created secret $secretName in Key Vault $keyVaultName, hence cleaning the variable from the keyvault"
        az pipelines variable-group variable delete --group-id $groupID --name "$name --yes"
      fi
    fi
    done < "$(keyvault.secureFilePath)"

In this case, the variable value will not be exposed during the pipeline run. And the secure file will be automatically deleted(On agent machine) after Pipeline run completing.

For more detailed info, you can refer to this doc: Use secure files

create them in key vault only if the secret value is different(if existing)

You can add the Azure CLI: Retrieve a secret from Key Vault to bash loop to compare secret value and current variable value.

Update:

I can reproduce the same issue when using the same bash script.

The cause of the issue is that when we use the Azure CLI to get the existing secret value. it will automatically add \r at the end of the value.

This line:

existing_secretvalue=$(az keyvault secret show --name "$name" --vault-name "$(${{ variables.podkv }})" --query "value")    

For example:

To solve this issue, you can add --output tsv to format the output of the secret value.

For example:

az keyvault secret show --vault-name "$(${{ variables.podkv }})" --name "$name" --query value --output tsv

Update2:

it will not pick the value for variable which is having special character "=" in its value.

To solve this issue, you can use the following bash script to read the varaibles in file.

while IFS=$'=' read -r key values;
do    
  name=$key
  value=$values   
  echo $name
  echo $value
done < "$(keyvault.secureFilePath)"

Result:

The echo command is used to confirm if the value is correct. You can remove it in your bash script.