azurepowershellazure-devopsazure-data-factoryazure-devops-rest-api

How to retrieve a List of Pipelines in azure data factory for a Specific Branch?


I need to retrieve a list of pipelines in Azure Data Factory (ADF) and are associated with a specific branch in a specific repository within Azure DevOps.

Azure DevOps Pipelines

Here’s what I’ve tried so far:

Azure DevOps REST API: I used the following API call in powershell to retrieve details from the branch in the repository, as the pipeline details are stored inside ADF/pipelines, but this is not giving me the expected result:

GET https://dev.azure.com/{organization}/{project}/_apis/git/repositories/{repositoryId}/items?path={path}&scopePath={scopePath}&recursionLevel={recursionLevel}&includeContentMetadata={includeContentMetadata}&latestProcessedChange={latestProcessedChange}&download={download}&$format={$format}&versionDescriptor.versionOptions={versionDescriptor.versionOptions}&versionDescriptor.version={versionDescriptor.version}&versionDescriptor.versionType={versionDescriptor.versionType}&includeContent={includeContent}&resolveLfs={resolveLfs}&api-version=5.0

I got this response from the above request, but my expectation was to retrieve the list of adf pipelines triggered from a specific branch.

count : 1
value : {@{objectId=xxxxxx; gitObjectType=tree; 
        commitId=xxxxxx; path=/ADF/pipeline; isFolder=True; url=https://xxxx.xxxx.xxx
        /xxxx/xxxxxxxx/_apis/git/repositories/xxxxxx/it
        ems?path=%2FADF%2Fpipeline&versionType=Branch&version=Main&versionOptions=None}}

PowerShell Module: I used Get-AzDataFactoryV2Pipeline to list pipelines in the Data Factory:

Get-AzDataFactoryV2Pipeline -ResourceGroupName <ResourceGroupName> -DataFactoryName <DataFactoryName>

But, this cmdlet returns pipelines in the live mode in the Data Factory environment, not the Devops pipelines.


Solution

  • As per the json files screenshot, they are ADF pipelines actually. In DevOps git repo, it will have the name in the json file as below:

    enter image description here

    If you want to get the ADF pipeline name from DevOps git for specific branc, eg: Develop, you can checkout the branch, and get the name value from the json.

    Yaml sample below:

    pool:
      vmImage: Windows-latest
    
    steps:
    - checkout: self
      persistCredentials: true                 # keep credential for git command latter.
    
    - powershell: |
            git fetch --all
            git checkout Develop                  # for Develop branch
            # Get all JSON files in the "ADF/pipeline" folder
            if (Test-Path -Path "ADF/pipeline") {
              $jsonFiles = Get-ChildItem -Path "ADF/pipeline" -Filter *.json -Recurse
              foreach ($file in $jsonFiles) {
                  $jsonContent = Get-Content -Path "ADF/pipeline/$file" -Raw | ConvertFrom-Json
                  $pipelineName = $jsonContent.name
                  Write-Output "Branch: Develop, PipelineName: $pipelineName"       # output branch name and pipeline name.
              }
             } else {
                Write-Output "Path 'ADF/pipeline' not found, skipping..."
            }
    

    It can get the pipeline name:

    enter image description here