How can I to delete all files and all subdirectories from a folder, but not delete the folder itself in Azure Fileshare using ADF? Basically at the end of the delete, there should be an empty folder.
I tried using the delete activity with Filepath in Dataset option, but it deletes the entire source folder.
I also tried using the Wildcard file path = "*", but it only deletes files and cannot delete subdirectories.
This is my folder structure. I want to delete files and subfolders marked with a red diagonal line. You can use the folder EXPORT/IN/REF-A as an example.
I know I can use Foreach and Get metadata, but because the folder data I need to delete is an array and ADF does not allow nested for activity, I would need to create an additional pipeline, which would lead to additional costs, and my client is not comfortable with this approach.
Appreciated for for all your assistance.
You can follow below approach to achieve your requirement:
Here is the structure of the folder:
Use get metadata activity to list the items in REF-A folder with field list Child items with below dataset:
You will get output as shown below:
On success of meta data activity add foreach activity with items @activity('Get Metadata1').output.childItems
. Add one delete activity to delete sub folders with dataset parameter directorypath for directory with value @concat('files/d1/EXPORT/IN/REF-A/',item().name)
as show below:
src1 dataset:
Add another delete activity to delete files in REF-A folder with dataset parameter filename for file name with value @item().name
as shown below:
src2 dataset:
Debug the pipeline, it will delete all files and sub folders inside REF-A folder, and it will remain REF-A folder as empty as shown below:
Here is the pipeline Json for your reference:
{
"name": "pipeline1",
"properties": {
"activities": [
{
"name": "Get Metadata1",
"type": "GetMetadata",
"dependsOn": [],
"policy": {
"timeout": "0.12:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"dataset": {
"referenceName": "b1",
"type": "DatasetReference"
},
"fieldList": [
"childItems"
],
"storeSettings": {
"type": "AzureFileStorageReadSettings",
"enablePartitionDiscovery": false
},
"formatSettings": {
"type": "BinaryReadSettings"
}
}
},
{
"name": "ForEach1",
"type": "ForEach",
"dependsOn": [
{
"activity": "Get Metadata1",
"dependencyConditions": [
"Succeeded"
]
}
],
"userProperties": [],
"typeProperties": {
"items": {
"value": "@activity('Get Metadata1').output.childItems",
"type": "Expression"
},
"isSequential": true,
"activities": [
{
"name": "Delete1",
"type": "Delete",
"dependsOn": [],
"policy": {
"timeout": "0.12:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"dataset": {
"referenceName": "src1",
"type": "DatasetReference",
"parameters": {
"directorypath": {
"value": "@concat('files/d1/EXPORT/IN/REF-A/',item().name)",
"type": "Expression"
}
}
},
"enableLogging": false,
"storeSettings": {
"type": "AzureFileStorageReadSettings",
"recursive": true,
"enablePartitionDiscovery": false
}
}
},
{
"name": "Delete2",
"type": "Delete",
"dependsOn": [
{
"activity": "Delete1",
"dependencyConditions": [
"Succeeded"
]
}
],
"policy": {
"timeout": "0.12:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"dataset": {
"referenceName": "src2",
"type": "DatasetReference",
"parameters": {
"filename": {
"value": "@item().name",
"type": "Expression"
}
}
},
"enableLogging": false,
"storeSettings": {
"type": "AzureFileStorageReadSettings",
"recursive": true,
"enablePartitionDiscovery": false
}
}
}
]
}
}
],
"annotations": []
}
}