azure-data-factory

Azure Data Factory (ADF) validate data types from incoming csv


My use case is to pick a .csv file which comes from a datalake container, validate the datatypes of the data included in the .csv (if data types are not valid fail the pipeline) and finally load the data into a DB table.

Within a dataflow, in a datasource transformation, in the projection tab, I did a detect data type and it accurately it discovers the data type of some fields.

enter image description here


I also picked this option of validate schema which is supposed if the data type of data does not match the projection.

enter image description here


To artificially test it, I uploaded a .csv with str type where an int should be expected.

The dataflow did NOT fail though.

Any ideas on how to meet the above-mentioned requirement?


Solution

  • Validate Schema option checks the structure of source data and sink data if its mismatch it will throw an error.

    Any ideas on how to meet the above-mentioned requirement?

    To detect if column mismatched is you can use the assert transformation and Fil the data flow if datatype mismatched.

    For String  expression: iif(isInteger(Colname),false(),true())
    For Integer  expression: iif(isInteger(colname),true(),false())
    

    enter image description here