azureazure-data-factoryetldata-warehousedata-lakehouse

ETL / ELT pipelines - Metainformation about the pipeline


how do you add metainformation about the used ETL / ELT code (and version of this ELT code) to the produced sink files / tables?

Do u consider it as required to have information like "PipelineID" or "DataProductionTime" in the targetfolder?


Solution

  • how do you add metainformation about the used ETL / ELT code (and version of this ELT code) to the produced sink files / tables?

    You can do it using the pipeline dynamic content and additional column in copy activity.

    This is my source file:

    enter image description here

    In source of copy activity use additional column.

    enter image description here

    In ADF you can find meta information about the pipeline in System variables of dynamic content.

    enter image description here

    Target file in sink folder:

    enter image description here

    Do u consider it as required to have information like "PipelineID" or "DataProductionTime" in the targetfolder?

    All this depends on your requirement like if you want to know by which pipeline you are getting data you can use this.