azureazure-data-factory

Data Filtering Issues

I'm trying to filter in the body request in Copy Activity, but the issue is that it's not filtering the data. When I do this in Postman, the data gets filtered. Why is this so? Can someone please tell me what I am doing wrong?

 {
"Filters": [    
{
  "Route": "Production.HasBeenAddedBy",
  "Value": 5,
  "Operation": "NotEquals"
}
],
"Paging": {
"RecordsPerPage": 100,
"RequestedPage": 1
},
"ResponseBlocks": [

    "VacancyID",
"StartDate",
"EndDate",
"candidate"

] }

Solution

  • I have tried the above filter query, but I am facing the same issue. You can try Dataflow as a workaround if none of your source and sink are not on-premises.

    Give your source data as source in the dataflow. In between the source and sink, you can use transformations like select and filter to get the required data from source.

    enter image description here

    Use the select transformation to select your inner objects as columns. Then, use filter transformation to filter out the required data from the columns. You can make use of Dataflow expressions to build the correct expression as per your input data.

    You can use REST API dataset as the sink of the Dataflow.

    enter image description here

    Set the pagination here and call the dataflow from the pipeline using dataflow activity.