sql-server-2008ssiscomparison

Conditional split based on array variable


I need something like a T-SQL IN statement to filter records in a conditional split based on an array variable (or something similar)

I need to have a list of items that a column can be filtered on.


Solution

  • As Filip has indicated, there is no IN operator in the expression language. I did come up with some options though as I thought this sounded like an interesting problem.

    My long analysis is on my blog: Filter list in SSIS

    Conditional split

    If you can transform your list of values into a delimited string, then you can use FINDSTRING and the current value to determine whether it's in the list. This provided the best throughput for my testing scenario. (FINDSTRING(@[User::MyListStr], [MyColumn],1)) > 0

    Script task

    I had assumed using a List in a script task to determine membership would provide the best performance but I was wrong. Row.IsInList = MyListObj.Contains(Row.MyColumn);

    Lookup/Cache Connection Manager

    The third approach I had come up with was dumping the list into a Cached Connection Manager and then using that in a lookup task. I thought this was the easiest to conceptualize and maintain but the performance was lacking.

    Conclusion

    For this problem domain, the FINDSTRING approach was the most efficient, by a considerable margin. The other two approaches consistently averaged a throughput of within 7 rows per millisecond of each other. I did find it interesting that the standard deviation of the FINDSTRING approach fluctuated so much. While this box is older and slower, there was not a considerable amount of activity going on during the package executions.