sqlazure-sql-databasedenormalized

Data Migration from Denormalized to Normalized Versions of Azure SQL


I have an exact copy of DB2 data in a denormalized instance of Azure SQL. As new transactions are added to that denormalized database in the cloud, I would like that data to then flow to a normalized version of Azure SQL. Can I accomplish that using Azure Data Factory?


Solution

  • Don't use triggers. Use Change Data Capture (CDC).

    It's a rather new feature, but promising. Try out new features. Note there are some requirements like having a S3 tier. There are loads of other things, from ADF to Functions. ADF even has a CDC function of its own: https://learn.microsoft.com/en-us/azure/data-factory/concepts-change-data-capture is helpful to CDC to other destinations and do transformations.

    You can even stream your data, with something like Spark structured streaming or ASA. The cheapest option will always be to write your own delta loads.