We are ingesting data from different resources to our data lake with ADF pipelines and have hundreds of pipelines (under 1 resource group). Cost per pipeline is already activated and I'm embedding Azure Portal cost data to a PowerBI Dashboard and can track cost per pipeline.
However, due to pipeline owners are not stated in there, I need to find owners of pipeline each time.
To overcome this issue, I tried to add annotation but couldnt see the pipeline's cost in cost management.
How can I tag pipelines in ADF (I have same issue for synapse too) and split the cost of each pipeline by their tags?
Thanks All!
Azure Data Factory has a feature to split service cost per pipeline. So, you can breakdown the cost to pipeline level, but there is no tagging feature is provided by Azure to seperate cost on Azure Cost Management or on the export of Azure cost data for dashboards.
There is only "Annotation" structure on ADF pipeline provided and its used to monitoring purpose. If you have hundreds of pipelines and would like to filter in ADF portal>Monitoring its useful. Annotations can created on ADF Portal>pipelines. Click a pipeline and at the right top click "Properties" and "+ New" under annotations.
Here is the details : https://learn.microsoft.com/en-us/azure/data-factory/concepts-annotations-user-properties
For the Synapse, there is no feature to enable cost per pipeline and also for the tagging for pipelines.