azure-data-factorymicrosoft-fabric

Azure Data Factory Copy Activity appends table in sink


I have a problem with a copy activity in Azure Data Factory. The setup is the following. Pipeline overview

The first copy data activity has Salesforce as a source. It receives one date from a query and writes the result in a Fabric Lakehouse table. The table action is set to "Overwrite," which means the table will never grow, and only one row will always be overwritten.

The second copy data action receives the same value from the table and sinks it in another Fabric Lakehouse table with the same settings as the first one. However, this one appends the table instead of overwriting it. Here's the result of the sink table:

Sink table 2

Does anyone know why the "overwrite" doesn't work here?


Solution

  • This seems to be a bug in Azure Data Factory. When I create a new pipeline with just one "Copy data" activity and set everything up as I described (read from one table, write into another, and option "Overwrite"), this is what the resulting JSON looks like. It adds "OverwriteSchema" instead of "Overwrite". I changed it manually in the JSON, and now everything works as expected. enter image description here

    Now, I manually changed the "tableActionOption" to "Overwrite," and everything works as expected.

    enter image description here