azureazure-data-factoryetllinked-service

Azure data factory data flow test connection Spark job failes

Im using Linked service to connect the Azure SQL database to the Azure data factory data flow.

In the Azure data factory Linked service connection gets established successfully. enter image description here

However, when I add my source dataset using a linked service to the data flow, there the test connection fails. enter image description here

Below are the error messages:

Spark job failed: {
"text/plain": "{\"runId\":\"94405efb-4fd6-449e-aa7f-e4396f1db87f\",\"sessionId\":\"93dd501b-ac03-4c93-8773-36058e06b8a0\",\"status\":\"Failed\",\"payload\":{\"statusCode\":400,\"shortMessage\":\"com.microsoft.dataflow.broker.MissingRequiredPropertyException: server is a required property for AzureSqlDatabase1.\\ncom.microsoft.dataflow.broker.PropertyNotFoundException: Could not extract value from AzureSqlDatabase1\",\"detailedMessage\":\"Failure 2024-05-17 07:10:36.828 failed DebugManager.processJob, run=94405efb-4fd6-449e-aa7f-e4396f1db87f, errorMessage=com.microsoft.dataflow.broker.MissingRequiredPropertyException: server is a required property for AzureSqlDatabase1.\\ncom.microsoft.dataflow.broker.PropertyNotFoundException: Could not extract value from AzureSqlDatabase1\"}}\n"
} - RunId: 94405efb-4fd6-449e-aa7f-e4396f1db87f

When I googled the above error most of the forums said to recreate the Linked service. In this StackOverflow thread it asks to increase the compute size of the Data Flow Integration which also does not work.

Is there any better solutions, or what exactly cause this issue?

Solution

  • I had to create a new Linked Service, this time using the "legacy" version and not the "recommended". Add my server name and db name and test. Then publish the changes with that new LS. (I might have been able to use the existing, so long as I published after) Then change my sources to use that new Linked Service (I was getting the same error as you but in CDC using the Linked Service direct). I then no longer got that error. I think there might be a bug (seen similar posts on MS) and using Legacy in the Linked Service avoids it Here is my Linked Service config using Legacy