azureazure-data-factory

Unable to read Parquet files(Created from Databricks Job/Notebook) from Azure Data Factory


I am getting error while I am reading Parquet file created by Databricks on ADLS. While I read these files using Databricks it works perfectly fine and I am able to read and write data into these files from Databricks. However with DataFactory it is giving below error.

Error: Parquet file contained column 'txn', which is of a non-primitive, unsupported type.

However there is no txn column create by me from Databricks.


Solution

  • This error mainly happens because of unsupported data type. When you pass to the column in parquet file make sure you are using the supported datatype.

    Supported datatype mapping for parquet file refer this Microsoft document .