azure-blob-storageazure-data-factorylast-modified

Get last modified folder using Azure Data Factory


I am trying to get the directory (i.e. /DATE=2023-01-03/ID=333) that has been modified within the last 5 minutes, then pass that folder path as a value to the source path of a Copy data activity. I've tried using Get Metadata and For Each activities, but it didn't work the way I expected.

Here is what the folder structure looks like:

silver/

|---------DATE=20230101/
|---------|---------ID=111
|---------|---------|---------part-1.c000.snappy.parquet
|---------|---------|---------part-2.c000.snappy.parquet
|---------|---------|---------committed_123    
|---------DATE=20230103/
|---------|---------ID=222
|---------|---------|---------part-1.c000.snappy.parquet
|---------|---------|---------part-2.c000.snappy.parquet
|---------|---------|---------committed_123
|---------|---------ID=333
|---------|---------|---------part-1.c000.snappy.parquet
|---------|---------|---------part-2.c000.snappy.parquet
|---------|---------|---------committed_123
|---------_SUCCESS

In this example, I'd like to get the folder path (/DATE=2023-01-03/ID=333), which was last modified within last 5 minutes, then pass that path as a value for source path of a Copy activity. However, _SUCCESS file also gets updated and is most likely to appear in the top of the last modified datetime list, so I'd like to exclude _SUCCESS and just get the last modified folder path passed to the Copy activity.


Solution

  • enter image description here

    enter image description here

    @equals(item().type,'Folder')
    

    enter image description here

    enter image description here