azure-data-factorydelta

How to bulk process a Database Delta sql file in Azure Data Factory to update a MySQL database


I have a database delta .sql file that contains updates to table in a MySql database. I understand that data factory can execute sql statements and I could parse the .sql file a csv split by delimiter of ';' to get the sql statements. However my delta file contains a large number of statements and looping this in ADF would mean it would call such activities potentially hundreds of thousands of times - is it possible to execute statements in bulk?

Reference to an implantation with iteration sql statements: How to process a Database Delta sql file in Azure Data Factory to update a MySQL database

or if there not an efficient way to update through iterating the .sql file? Is the only sensible approach a function app to execute the sql?

Many Thanks.


Solution

  • You can run all the queries in .sql file in a single batch itself. Below is the approach.

    @concat(activity('Lookup1').output.value[0].Prop_0,'select 1 as col1')
    

    if the select 1 as col1 is not concatenated with the output of lookup activity, then ADF will throw error. Because Lookup activity expects to return data.