I want to concat every column within a table to one string separated with a separator. In PySpark this would look like this:
concat_ws("#?", *columns)
What would be the equivalent in data flow expression language?
concatWS('#?', columns())
won't work. I get the following error:
"Error: Function 'concatWS' argument 3 should be StringType but was BinaryType"
Spark obviously implicitly converts every column to a string
Even tried this, but does not work either:
concatWS('#?', toString(byNames(columnNames())))
Thanks, Jan
Your expression is correct and working fine.
concatWS('#?', columns())
If you still not getting expected results you can use the below alternate expressions.
concatWS('#?', toString(columns()))
Or
rtrim(toString(reduce(map(array(columns()),toString(#item)+'#?'),'', #acc + toString(#item), #result)),'#?')
Below is the columns i am having.
Output: