I have a Azure Synapse pipeline which we migrated from DEV environment to our Pre Prod environment. When I execute it via the debug option, the pipeline executes fine. However if I trigger it via 'Trigger Now' or via setting up a trigger the job fails at a particular data flow. This particular data flow uses a cache sink; so I am unable to set 'verbose logging'.
The manual execution and the trigger use the same run time.
Following is the error message that I have been receiving:
{ "errorCode": "BadRequest", "message": "The request failed with status code '"BadRequest"'.", "failureType": "UserError", "target": "Dataflowname ", "details": "" }
Solved!!!
We posted this question to Microsoft and they looked at this and identified the issue. The Integration Run Time names across our DEV and Pre Prod had slight differences. The name of the IR, which was embedded in the pipelines definitions (which got migrated to Pre Prod), had a capital alphabet in DEV and Pre Prod had it in lower case. Once the names of the IR were rectified, the jobs were working fine when triggered.
MS advised that when you run it manually via the Debug option, the differences in the name of the IR doesnt matter. However when its triggered the IR names have to be consistently across all the environments, if the names are embedded in the pipeline definitions.