How to check with ADF if the files located in the SFTP server are also in the Azure storage account and if not to copy the files missing

I'm new to Azure Data Factory (ADF), and my company has asked me to design a pipeline that checks whether files from an SFTP server exist in an Azure Storage Account. If any files are missing, they should be copied automatically.

The goal is to verify that the main pipeline has correctly imported all data; if not, this secondary pipeline will handle missing files.

Files are stored in different folders with distinct formats:

1. Import folder: 2 files (fileName_yyyyMMdd)
2. Export/folder1: 8 files:
   • 5 files (fileName_dd-MM-yyyy)
   • 3 files (fileName_yyyy-MM-dd)

I need to set up ADF to dynamically check file existence and process the missing ones accordingly. In the storage account, the files transferred are in a folder, unorganized (files land there waiting to be send to another layer to be processed).

From what i understood I need to use a Get Metadata then a Foreach to filter each file and inside the Foreach a If Condition to check if the file is inside the Storage Account and when the If Condition is true meaning the file is missing then copy the data. But I am not really sure if this is the good way to do that. Anyone can help me out here?

Thanks

No need of a for each to get the list of missing files.

You can use 2 get meta data activities : 1 to get list of files in SFTP and another to get list of files in blob..

Then you can use Filter activity to get the list of files present in SFTP and not in Blob and that should be in provided to for each activity to copy those files into blob dynamically