We are encountering an issue with the Copy activity in Azure DataFactory. Our data source is Oracle, and we use dynamic range partitioning. The data is then saved as parquet files.
Sometimes the process completes successfully, but at other times we receive the following error:
Failure happened on the 'Sink' side. ErrorCode=UserErrorFileNotFound, 'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException, Message=ADLS Gen2 operation failed for: Operation returned an invalid status code 'NotFound'. Account: 'XXXX'. FileSystem: 'XXXX'. Path: 'PATH_TO_PARQUET_FILE/oracle_query_TABLE_ID_91922989_122230649_00003.parquet'. ErrorCode: 'PathNotFound'. Message: 'The specified path does not exist.'. RequestId: 'XXXXX'. TimeStamp: 'Sat, 03 Aug 2024 03:10:20 GMT.', Source=Microsoft.DataTransfer.ClientLibrary,'Type=Microsoft.Azure.Storage.Data.Models.ErrorSchemaException, Message=Operation returned an invalid status code 'NotFound', Source=Microsoft.DataTransfer.ClientLibrary,'
We suspect that the partitions are changing after calculation, as indicated by the start and end ID in the file name.
Do you have any advice on how to address this error?
Thank you.
FYI, i enabled staging in the copy activity, it looks like it's working.