azureazure-devopsazure-data-factory

How to automate Azure Data Factory (ADF) credential updates from development to production using Azure DevOps?

I'm working with Azure Data Factory (ADF) and need to automate the promotion of credentials (e.g., Linked Service connection strings, secrets) from the development environment to production using Azure DevOps. I want to create a release pipeline that:

What is the best way to achieve this using ARM templates and Azure DevOps? I would also like to understand how to override Key Vault secrets between environments during deployment.

Any guidance or step-by-step examples on how to set up this release pipeline would be appreciated!

What I've tried so far:

I have this setup in DevOps to deploy pipelines. enter image description here

Solution

  • Not sure about your specific configuration of ADF, here is an example for deploying from one env to another.

    1. Config git for your DEV ADF to use Azure repo. See the details from Connect to a Git repository.

    2. Store sensitive information in Azure Key Vault.

    3. Create a release pipeline in Azure DevOps.

      For artifact:

      • Source type: Azure Repo
      • Default branch: adf_publish

      enter image description here

      For stages:

      Add two stages, one for Dev and another for Prod. enter image description here

      For tasks:

      • Use Azure Key Vault task to download secrets inside it. enter image description here

        Azure subscription: An ARM service connection to access your Azure Key Vault.

        Key vault: Select the Key Vault storing the sensitive information for your current env.

      • Use ARM template deployment task to deploy your ADF.

        Fill in the parameters above Template based on your actual conditions.

        Template: Use the Browser Template button to select the template from the artifact. For example, ARMTemplateForFactory.json.

        Template parameters: Select your template parameter, ARMTemplateParametersForFactory.json.

        enter image description here

        Override template parameters: Click the three-dot button after "Override template parameters" to override parameters.

        • You can reference secrets in Azure Key Vault directly in your pipeline using the format $(SecretName), for example, -AzureSqlDatabase1_password $(DB-Secret-Prod) in the screenshot below.
        • For different env, use different secrets/keyVaults.
        • If you already linked Azure Key Vault to store sensitive information in ADF, you should parameterize the key vault name and override the key vault here instead of the secrets inside it. Learn about parameters from Parameters in ARM templates.

        enter image description here

    The above example may not be 100% applicable to you, but the idea of ​​integrating with Azure Key Vault and deploying ADF to different environments is the same. For any credential in the template, parameterize and override it.