azuresnowflake-cloud-data-platformazure-data-factoryazure-data-lakeazure-data-lake-gen2

Azure Data Factory V2 Copy Activity is not cleaning/deleting staged files/containers after copying data to the destination (Like: Snowflake)


Azure Data Factory V2 Copy Activity is not cleaning/deleting staged files/containers after copying data to the destination (Like: Snowflake).

Is this a bug? In my ADF, the container level is set for the staging area and I end up with multiple GUID containers. Some containers are even having metadata files.

According to Microsoft, all the temporary files/folders/containers created by Azure Data Factory in the Staging Area should be automatically deleted by the ADF after the Copy is successful.

Please guide.


Solution

  • According to Microsoft, all the temporary files/folders/containers created by Azure Data Factory in the Staging Area should be automatically deleted by the ADF after the Copy is successful.

    For stagging we have 2 options for linked service:

    1. Azure Blob Storage.
    2. Azure Data Lake Service.

    Blob storage After stagging: enter image description here

    ADLS After stagging: enter image description here

    Container is not get deleted in either of the storage accounts.