azure-data-factoryincremental-load

How to implement incremental load on ADF

We have 2 folders in adls source and destination. source folder has 4 file and destination has 3. How can we implement the Increment load in ADF. What all activities we have to perform to achieve incremental load in ADF.

My approach :

1) use getmetadata activity parallelly for both source and 

destination folder and pass child item on field list.
2) store the immediate result on set variable activity parallelly for both source and destination.
3) Use If condition and pass condition if(not(contains(destination_set_varriable,set_varriable) then use copy activity
Solution

  • You can follow below procedure to achieve your requirement:

    Use two get metadata activities to get files in source and destination with Child items field list, you can find outputs of get metadata as shown below:

    Source:

    enter image description here

    Destination:

    enter image description here

    Add filter activity on success of metadata activities with source child items as items @activity('source').output.childItems and give condition @not(contains(activity('destination').output.childItems,item())) as shown below:

    enter image description here

    You will get filter output activity as shown below:

    enter image description here

    Add copy activity to filter activity, take source with dataset parameter file_name for file name with value @activity('Filter1').output.Value[0].name as source and add sink dataset, debug the pipeline, the file which is not there in destination will copy to destination successfully as shown below:

    enter image description here