azureazure-blob-storageoracle-adfdata-pipeline

Azure Data Factory Copy Data HTTP source zip file and sink as azure blob storage, asking to create two copy activities to work around it


I am getting this error message when trying to copy data from a HTTP source and my sink destination is an azure blob storage

I have a simple setup. I am visting the URL anonymously as I do not require any cookies or authentication to access the download from AWS and I want to download the zip contents. The content on the zip contain multiple files and different file types, i want to keep these original file types and use copy data to get them to Azure Blob Storage as an extracted zip file

my sink link service is of type Azure Blob Storage.

Im not sure why it's difficult to simply copy a zip file to storage account?

Error message:

ErrorCode=UserErrorSourceNotSeekable,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Your HttpServer source can't support random read which is requied by current copy activity setting, please create two copy activities to work around it: the first copy activity binary copy your HttpServer source to a staging file store(like Azure Blob, Azure Data Lake, File, etc.), second copy activity copy from the staged file store to your destination with current settings.,Source=Microsoft.DataTransfer.ClientLibrary,'

example url https://s3.amazonaws.com/rds.nsrl.nist.gov/RDS/rds_2023.09.1/RDS_2023.09.1_modern_minimal_delta.zip

I am passing the rds_2023.09.1/RDS_2023.09.1_modern_minimal_delta.zip as output from my function and using it as relative url for the copy data

sink 1.1

sink 1

source

overview


Solution

  • I want to keep these original file types and use copy data to get them to Azure Blob Storage as an extracted zip file.

    Use the below compression in source data set to achieve your requirement:

    Source dataset:

    enter image description here

    Sink dataset:

    enter image description here

    I have tried same with your data and you can I got expected results:

    enter image description here