azureazure-data-factoryazure-files

How to delete files and subdirectories from folder but not that folder itself with ADF


How can I to delete all files and all subdirectories from a folder, but not delete the folder itself in Azure Fileshare using ADF? Basically at the end of the delete, there should be an empty folder.

I tried using the delete activity with Filepath in Dataset option, but it deletes the entire source folder. Filepath in Dataset

I also tried using the Wildcard file path = "*", but it only deletes files and cannot delete subdirectories. Wildcard filepath

This is my folder structure. I want to delete files and subfolders marked with a red diagonal line. You can use the folder EXPORT/IN/REF-A as an example.folder-structure

I know I can use Foreach and Get metadata, but because the folder data I need to delete is an array and ADF does not allow nested for activity, I would need to create an additional pipeline, which would lead to additional costs, and my client is not comfortable with this approach.

Appreciated for for all your assistance.


Solution

  • You can follow below approach to achieve your requirement:

    Here is the structure of the folder:

    enter image description here

    Use get metadata activity to list the items in REF-A folder with field list Child items with below dataset:

    enter image description here

    You will get output as shown below:

    enter image description here

    On success of meta data activity add foreach activity with items @activity('Get Metadata1').output.childItems. Add one delete activity to delete sub folders with dataset parameter directorypath for directory with value @concat('files/d1/EXPORT/IN/REF-A/',item().name) as show below: src1 dataset:

    enter image description here

    enter image description here

    Add another delete activity to delete files in REF-A folder with dataset parameter filename for file name with value @item().name as shown below:

    src2 dataset:

    enter image description here

    enter image description here

    Debug the pipeline, it will delete all files and sub folders inside REF-A folder, and it will remain REF-A folder as empty as shown below:

    enter image description here

    Here is the pipeline Json for your reference:

    {
        "name": "pipeline1",
        "properties": {
            "activities": [
                {
                    "name": "Get Metadata1",
                    "type": "GetMetadata",
                    "dependsOn": [],
                    "policy": {
                        "timeout": "0.12:00:00",
                        "retry": 0,
                        "retryIntervalInSeconds": 30,
                        "secureOutput": false,
                        "secureInput": false
                    },
                    "userProperties": [],
                    "typeProperties": {
                        "dataset": {
                            "referenceName": "b1",
                            "type": "DatasetReference"
                        },
                        "fieldList": [
                            "childItems"
                        ],
                        "storeSettings": {
                            "type": "AzureFileStorageReadSettings",
                            "enablePartitionDiscovery": false
                        },
                        "formatSettings": {
                            "type": "BinaryReadSettings"
                        }
                    }
                },
                {
                    "name": "ForEach1",
                    "type": "ForEach",
                    "dependsOn": [
                        {
                            "activity": "Get Metadata1",
                            "dependencyConditions": [
                                "Succeeded"
                            ]
                        }
                    ],
                    "userProperties": [],
                    "typeProperties": {
                        "items": {
                            "value": "@activity('Get Metadata1').output.childItems",
                            "type": "Expression"
                        },
                        "isSequential": true,
                        "activities": [
                            {
                                "name": "Delete1",
                                "type": "Delete",
                                "dependsOn": [],
                                "policy": {
                                    "timeout": "0.12:00:00",
                                    "retry": 0,
                                    "retryIntervalInSeconds": 30,
                                    "secureOutput": false,
                                    "secureInput": false
                                },
                                "userProperties": [],
                                "typeProperties": {
                                    "dataset": {
                                        "referenceName": "src1",
                                        "type": "DatasetReference",
                                        "parameters": {
                                            "directorypath": {
                                                "value": "@concat('files/d1/EXPORT/IN/REF-A/',item().name)",
                                                "type": "Expression"
                                            }
                                        }
                                    },
                                    "enableLogging": false,
                                    "storeSettings": {
                                        "type": "AzureFileStorageReadSettings",
                                        "recursive": true,
                                        "enablePartitionDiscovery": false
                                    }
                                }
                            },
                            {
                                "name": "Delete2",
                                "type": "Delete",
                                "dependsOn": [
                                    {
                                        "activity": "Delete1",
                                        "dependencyConditions": [
                                            "Succeeded"
                                        ]
                                    }
                                ],
                                "policy": {
                                    "timeout": "0.12:00:00",
                                    "retry": 0,
                                    "retryIntervalInSeconds": 30,
                                    "secureOutput": false,
                                    "secureInput": false
                                },
                                "userProperties": [],
                                "typeProperties": {
                                    "dataset": {
                                        "referenceName": "src2",
                                        "type": "DatasetReference",
                                        "parameters": {
                                            "filename": {
                                                "value": "@item().name",
                                                "type": "Expression"
                                            }
                                        }
                                    },
                                    "enableLogging": false,
                                    "storeSettings": {
                                        "type": "AzureFileStorageReadSettings",
                                        "recursive": true,
                                        "enablePartitionDiscovery": false
                                    }
                                }
                            }
                        ]
                    }
                }
            ],
            "annotations": []
        }
    }