I wanted to automate the creation of a Azure Data Factory pipeline by using an already existing pipeline, retrieving the JSON and create a new one with small changes (mostly name of the pipeline).
At the end of the code I'm doing this:
Set-AzDataFactoryV2Pipeline -ResourceGroupName $resourceGroupName -DataFactoryName $dataFactoryName -Name $newPipelineName -File $jsonFilePath
The error message I receive is this:
Error Code: BadRequest
Error Message: Pipeline Entity not valid
Request Id: 115c7391-7882-41a8-953e-ec1a0b64dcb9
Timestamp (Utc):06/21/2024 14:21:15
This is the powershell code:
# Retrieve the existing pipeline
$pipeline = Get-AzDataFactoryV2Pipeline -DataFactoryName $dataFactoryName -ResourceGroupName $resourceGroupName -Name $existingPipelineName
# Convert Pipeline configuration to JSON
$pipelineJson = $pipeline | ConvertTo-Json -Depth 10
# Parse the JSON configuration
$pipelineConfig = $pipelineJson | ConvertFrom-Json
# Update the configuration parameters
$pipelineConfig.name = $newPipelineName
# Convert the updated configuration back to JSON
$updatedPipelineJson = $pipelineConfig | ConvertTo-Json -Depth 20
# Save the updated JSON to a file
$updatedPipelineJson | Out-File -FilePath $jsonFilePath -Encoding utf8
I also tried to download the naked existing JSON from a working pipeline and don't touch it at all and reuploded, but it failed. Also when I only change the name of the pipeline (to prevent having it 2 times) it does not work.
Where is my mistake? This is the JSON: I'm still not sure if my json is wrong? Technically (JSONLINT) its valid. Help appreciated.
{
"Name": "00_Master_Pipeline",
"Activities": [
{
"DataFlow": {
"AdditionalProperties": null,
"ReferenceName": "dataflow1",
"DatasetParameters": {
"source1": {},
"sink1": {}
},
"Parameters": {}
},
"Staging": {
"LinkedService": null,
"FolderPath": null
},
"IntegrationRuntime": null,
"Compute": {
"ComputeType": "General",
"CoreCount": 8
},
"TraceLevel": "Fine",
"ContinueOnError": null,
"RunConcurrently": null,
"SourceStagingConcurrency": null,
"LinkedServiceName": null,
"Policy": {
"AdditionalProperties": null,
"Timeout": "0.12:00:00",
"Retry": 0,
"RetryIntervalInSeconds": 30,
"SecureInput": false,
"SecureOutput": false
},
"AdditionalProperties": {
"typeProperties": {
"dataflow": {
"referenceName": "dataflow1",
"type": "DataFlowReference",
"parameters": {},
"datasetParameters": {
"source1": {},
"sink1": {}
}
},
"staging": {},
"compute": {
"coreCount": 8,
"computeType": "General"
},
"traceLevel": "Fine"
}
},
"Name": "Data",
"Description": "YYYY",
"State": null,
"OnInactiveMarkAs": null,
"DependsOn": [],
"UserProperties": []
}
],
"Parameters": null,
"Id": "/subscriptions/YYYYY/resourceGroups/YYYYY/providers/Microsoft.DataFactory/factories/YYYYYY/pipelines/00_Master_Pipeline",
"ETag": "ZZZ",
"ResourceGroupName": "YYY",
"DataFactoryName": "XXX"
}
Set-AzDataFactoryV2Pipeline
requires the correct ARM template of data factory pipeline. Converting the results of Get-AzDataFactoryV2Pipeline
command to Json is not same as the ARM template that defines the pipeline. In fact, there is no way to export the adf pipeline arm template through powershell command. Here is the feedback request link for that request.
To get the correct Json file, you can copy manually the pipeline Json by following the below steps.
Json code representation({}) symbol
as shown in the below image.
img:1 View the JSON code representation of pipeline.
Copy the Json of pipeline activities and modify the pipeline name and store it in the file.
Then you can import the template using ADF UI or using Set-AzDataFactoryV2Pipeline powershell command.
img:2 Importing through ADF UI.