azureazure-data-factoryazure-adf

Having access to Records that went to Sink from a Copy Activity


The source of my Copy activity is the result of calling a REST API and the Sink is a Azure SQL Table that I insert those records in it. Now I want to know what records we just got inserted in that Sink so I can do some update statement on those records. So my question is how can I know what went into Sink so I can now update those records.


Solution

  • Two methods come to mind

    1. If the table has a timestamp of when the records were inserted then you could use that value to know which were just inserted.

    2. Instead of putting the records directly in the final table put them in a staging table. Then you can either update as part of the insert to move them to the final table do the updates OR update them in staging table and then copy over to the final table. Just remember to truncate the staging table every run so it only has the new records.