Issue with Azure Data Factory Copy Activity: Transferring Data from Azure Storage Table to SQL Database

If you face issue while copying data from source Azure Storage table to SQL, where you see few columns are missing, this can be resolved by giving the first row with dummy values rather than they being null.

Because when you preview in ADF, you will not see the certain columns; after providing dummy values for the first row, you will see the columns in ADF copy activity after a refresh.

Hope this helps others

This is a known behavior in Azure Data Factory (ADF) when working with Azure Table Storage as a source. When ADF reads from Azure Table Storage, it infers the schema dynamically it looks at the rows to determine what columns (properties) exist. If first row contains the null value it will not reflect it.

• If some properties (columns) are missing or null in the first row, ADF assumes they don’t exist, and doesn't include them in the schema.

To resolve this Workaround is by inserting a dummy first row that has non-null values for all expected columns, you effectively “force” ADF to detect the full schema.

here is sample table:

ADF preview:

So, after this:

• Refresh the source preview in ADF.