My use case is to pick a .csv
file which comes from a datalake container, validate the datatypes of the data included in the .csv
(if data types are not valid fail the pipeline) and finally load the data into a DB table.
Within a dataflow, in a datasource
transformation, in the projection tab, I did a detect data type
and it accurately it discovers the data type of some fields.
I also picked this option of validate schema
which is supposed if the data type of data does not match the projection.
To artificially test it, I uploaded a .csv
with str
type where an int
should be expected.
The dataflow did NOT fail though.
Any ideas on how to meet the above-mentioned requirement?
Validate Schema option checks the structure of source data and sink data if its mismatch it will throw an error.
Any ideas on how to meet the above-mentioned requirement?
To detect if column mismatched is you can use the assert transformation and Fil the data flow if datatype mismatched.
For String expression: iif(isInteger(Colname),false(),true())
For Integer expression: iif(isInteger(colname),true(),false())