powershellazure-data-factory

Create Data Pipeline in Azure Data Factory - JSON is not accepted

I wanted to automate the creation of a Azure Data Factory pipeline by using an already existing pipeline, retrieving the JSON and create a new one with small changes (mostly name of the pipeline).

At the end of the code I'm doing this:

Set-AzDataFactoryV2Pipeline -ResourceGroupName $resourceGroupName -DataFactoryName $dataFactoryName -Name $newPipelineName -File $jsonFilePath

The error message I receive is this:

Error Code: BadRequest
Error Message: Pipeline Entity not valid
Request Id: 115c7391-7882-41a8-953e-ec1a0b64dcb9
Timestamp (Utc):06/21/2024 14:21:15

This is the powershell code:

# Retrieve the existing pipeline
$pipeline = Get-AzDataFactoryV2Pipeline -DataFactoryName $dataFactoryName -ResourceGroupName $resourceGroupName -Name $existingPipelineName

# Convert Pipeline configuration to JSON
$pipelineJson = $pipeline | ConvertTo-Json -Depth 10

# Parse the JSON configuration
$pipelineConfig = $pipelineJson | ConvertFrom-Json
# Update the configuration parameters
$pipelineConfig.name = $newPipelineName    

# Convert the updated configuration back to JSON
$updatedPipelineJson = $pipelineConfig | ConvertTo-Json -Depth 20

# Save the updated JSON to a file
$updatedPipelineJson | Out-File -FilePath $jsonFilePath -Encoding utf8

I also tried to download the naked existing JSON from a working pipeline and don't touch it at all and reuploded, but it failed. Also when I only change the name of the pipeline (to prevent having it 2 times) it does not work.

Where is my mistake? This is the JSON: I'm still not sure if my json is wrong? Technically (JSONLINT) its valid. Help appreciated.

{
  "Name": "00_Master_Pipeline",
  "Activities": [
    {
      "DataFlow": {
        "AdditionalProperties": null,
        "ReferenceName": "dataflow1",
        "DatasetParameters": {
          "source1": {},
          "sink1": {}
        },
        "Parameters": {}
      },
      "Staging": {
        "LinkedService": null,
        "FolderPath": null
      },
      "IntegrationRuntime": null,
      "Compute": {
        "ComputeType": "General",
        "CoreCount": 8
      },
      "TraceLevel": "Fine",
      "ContinueOnError": null,
      "RunConcurrently": null,
      "SourceStagingConcurrency": null,
      "LinkedServiceName": null,
      "Policy": {
        "AdditionalProperties": null,
        "Timeout": "0.12:00:00",
        "Retry": 0,
        "RetryIntervalInSeconds": 30,
        "SecureInput": false,
        "SecureOutput": false
      },
      "AdditionalProperties": {
        "typeProperties": {
          "dataflow": {
            "referenceName": "dataflow1",
            "type": "DataFlowReference",
            "parameters": {},
            "datasetParameters": {
              "source1": {},
              "sink1": {}
            }
          },
          "staging": {},
          "compute": {
            "coreCount": 8,
            "computeType": "General"
          },
          "traceLevel": "Fine"
        }
      },
      "Name": "Data",
      "Description": "YYYY",
      "State": null,
      "OnInactiveMarkAs": null,
      "DependsOn": [],
      "UserProperties": []
    }
  ],
  "Parameters": null,
  "Id": "/subscriptions/YYYYY/resourceGroups/YYYYY/providers/Microsoft.DataFactory/factories/YYYYYY/pipelines/00_Master_Pipeline",
  "ETag": "ZZZ",
  "ResourceGroupName": "YYY",
  "DataFactoryName": "XXX"
}
Solution

  • Set-AzDataFactoryV2Pipeline requires the correct ARM template of data factory pipeline. Converting the results of Get-AzDataFactoryV2Pipeline command to Json is not same as the ARM template that defines the pipeline. In fact, there is no way to export the adf pipeline arm template through powershell command. Here is the feedback request link for that request.

    To get the correct Json file, you can copy manually the pipeline Json by following the below steps.

    img:1 View the JSON code representation of pipeline.

    img:2 Importing through ADF UI.