azure-data-factory

Dataset parameter not available in Copy Data Activity (source tab)


I'm new to ADF, i want to pass a parameter created in dataset the copy data activity. However, dataset parameter is not coming through in the pipeline expression builder of source copy data activity. As a result, i had to create another parameter enter_new_file_name. Not sure where I'm going wrong. Eventually, I'd like to use dataset parameter in the source copy data activity tab without having to created new one.

enter_file_name parameter created for data set

[however, the enter_file_name parameter not available in copy data activity2


Solution

  • The dataset parameters are used for passing the file names or folder names dynamically during the pipeline run to the dataset.

    Here is how you need to use it to pass a sample file name sample1.csv. I have used Lookup activity, but the usage is same in both lookup and copy activity.

    After creating the dataset parameter from the parameters section, open the dynamic content of the file name in the dataset. Here, it will show the created parameter and click on it. Now, when you pass any value to the parameter, it's value will be given to the file name of the dataset.

    enter image description here

    Give the dataset to the activity in the pipeline. Here, it will ask to provide the value to the parameter. You can either directly provide your file name here or you can use the pipeline parameters. Also, you can use ADF dynamic expressions here, to provide the required value.

    Create pipeline parameter of same data type in the parameter's sections of the pipeline and don't provide any default value. Open the dynamic content of the dataset parameter and here it will show the created pipeline parameter. When you click on it, it will assign this parameter to the dataset parameter.

    enter image description here

    That means, whatever the value (file name) that you will give to the pipeline parameter during the pipeline run, it will be passed to the dataset parameter and from this it will be passed to the file name in the dataset, and it will give the data of that file.

    When you hit the debug, it will ask to provide value to the pipeline parameter. Give your file name here and click on ok.

    enter image description here

    Now, the pipeline will start it's run, and the activity will do its job. You can see the provided value in the input JSON of the activity.

    enter image description here

    You can go through this SO answer to understand more about dataset parameterization.