I have below scenario used in the current azure data factory pipeline

1. SFTP server, which will have file available in certain date - say between 1st and 10th day
2. ADF pipeline schedule to for these days - 1st to 10th day
3. The availability of file are not consistent - so pipeline have to check in every days - between these range

Problem
ADF pipeline run for unnecessary days - (even one the file already onboarded - say after 1st) and creating cost for other days

Expected:

Is there any way where can scheduled ADF pipeline to be trigger only when the file available in the source- SFTP?

Unfortunately, ADF does not support true event-based triggers for SFTP.

ADF event triggers only work with Azure Blob Storage, Azure Data Lake Storage Gen2 and Azure Event Grid-integrated services. Because SFTP does not emit events that ADF can consume.

To simulate event-based triggering for SFTP file availability, a robust and Azure-native workaround is to use Azure Blob Storage as an intermediary layer. This allows you to leverage native Blob-created events to trigger your ADF pipeline, only when a file is detected.

1. To connect your SFTP server to Azure Blob Storage, kindly refer - SFTP support for Azure Blob Storage

2. Now, we can create an event-based trigger in ADF that automatically starts the pipeline when a new file appears in Azure Blob Storage. This enables the pipeline to run only when new data arrives. For detailed guidance, refer to the documentation - Creating a trigger that runs a pipeline in response to a storage event