filterazure-data-factorydataverse

Issue with Filtering on Dataverse Source


I have a filter in a Azure data factory Data Flow. enter image description here

and(
    ptl_partner == '024D96D0-EF1A-4CD9-9C51-D4D53BEE69D3',
    statuscode == 2
)

This filter is very simple and is acting on a dataverse source. When I run this Data Flow, no data manages to get through the filter. However, when I run a SQL query identical to the filter against the source, it returns 54 rows.

select *
from ptl_matter m
where m.ptl_partner = '024D96D0-EF1A-4CD9-9C51-D4D53BEE69D3'
and statuscode = 2

I'm unsure what exactly the issue is. Is this a know problem with using dataverse as a source in data factory?

Removing the filter from the Data Flow allows data to pass through, so it's not a problem with anything else in the flow.

EDIT: The 4 columns I am interested in have the following data types:

Here's a preview of the 4 columns I'm interested in. I'm only showing these 4 as there's over 300 columns on the source table. enter image description here


Solution

  • The issue seems to because of ADF dataflow filter is case sensitive. And when you query the same in SQL database, it is case insensitive and thus it returns all 54 rows. To solve this, you can convert the value of ptl_partner field to upper case using toUpper() function and then filter using == operator. Otherwise, you can use equalsIgnoreCase() function to ignore the case while filtering. Below is the correct expression to give in filter transformation in dataflow

    and(
        ptl_partner <=> '024D96D0-EF1A-4CD9-9C51-D4D53BEE69D3',
        statuscode == 2
    )
    

    (or)

    and(
        equalsIgnoreCase(ptl_partner, '024D96D0-EF1A-4CD9-9C51-D4D53BEE69D3'),
        statuscode == 2
    )