azure-data-factoryazure-adf

ADF Copy only when a new CSV file is placed in the source and copy to the Container

I want to copy the file from Source to target container but only when the Source file is new (latest file is placed in source). I am not sure how to proceed this and not sure about the syntax to check the source file greater than target. Should i have to use two get metadata activity to check source and target last modified date and use if condition. i tried few ways but it didn't work. Any help will be handy

enter image description here

syntax i used for the condition is giving me the error

@if(greaterOrEquals(ticks(activity('Get Metadata_File').output.lastModified),activity('Get Metadata_File2')),True,False)

error message

The function 'greaterOrEquals' expects all of its parameters to be either integer or decimal numbers. Found invalid parameter types: 'Object'

Solution

  • You can try one of the Pipeline Templates that ADF offers.

    enter image description here

    enter image description here

    Use this template to copy new and changed files only by using LastModifiedDate. This template first selects the new and changed files only by their attributes "LastModifiedDate", and then copies them from the data source store to the data destination store. You can also go to "Copy Data Tool" to get the pipeline for the same scenario with more connectors.

    View documentation

    OR...

    You can use Storage Event Triggers to trigger the pipeline with copy activity to copy when each new file is written to storage.

    Follow detailed example here: Create a trigger that runs a pipeline in response to a storage event