azureazure-data-factorywebapiazure-data-lake-gen2

How to create a file with contents using ADLS GEN2 API Using ADF


I need to create a file in ALDS GEN2 using WEB API activity in ADF.

Below is the request i am using to create a file.

{
    "method": "PUT",
    "headers": {
        "x-ms-date": "Wed, 05 Mar 2025 08:25:13 GMT",
        "Content-Length": "1048",
        "x-ms-version": "2015-02-21",
        "x-ms-blob-condition-appendpos": "2097152 ",
        "x-ms-blob-condition-maxsize": "4194304"
    },
    "url": "https://sample.dfs.core.windows.net/dsa-f/saf/temp/dsa/archive/test_aud.done?comp=appendblock",
    "connectVia": {
        "referenceName": "dsadfesf",
        "type": "IntegrationRuntimeReference"
    },
    "body": "{Data_Count:2}",
    "authentication": {
        "type": "MSI",
        "resource": "https://storage.azure.com/"
    }
}

But i am getting below error and according to the documentation i have passed all the required values in header

{"error":{"code":"MissingRequiredHeader","message":"An HTTP header that's mandatory for this request is not specified.\nRequestId:safsf-ewj-sjcb-315b-cdsccdsc\nTime:2025-03-05T16:35:49.3616136Z"}}

I have referred following documentation https://learn.microsoft.com/en-us/rest/api/storageservices/append-block?tabs=microsoft-entra-id

Can anyone help me to resolve the above issue and create file with contents.

Authentication is System Assigned Managed Identity, please find the below screenshot

Note: In the above web api call i am trying to write the contents to existing file. If there any other way to write the contents to the file while creating itself, please help me with that.

enter image description here


Solution

  • To create the file with data, you need to create and append data is in three stages using this Path - Update API.

    First, you need to create file using Path - Create with these details in ADF web activity:

    URL - https://<storage_account_name>.dfs.core.windows.net/<container_name><folder_and_filepath>?resource=file&x-ms-version=2018-11-09
    
    Method - PUT
    
    Body - {}
    
    Authentication- System-assigned managed identity
    
    Resource - https://storage.azure.com/
    
    Headers:
    
    Content-type - application/octet-stream
    x-ms-date - @{formatDateTime(utcnow(),'r')}
    

    Then you need need to call the action=append to push it to the uncommitted buffer on the server e.g.

    URL - https://<storage_account_name>.dfs.core.windows.net/<container_name><folder_and_filepath>?action=append&position=0&x-ms-version=2018-11-09
    
    Method - PATCH
    
    Body - {Data_Count:2}
    
    Authentication- System-assigned managed identity
    
    Resource - https://storage.azure.com/
    
    Headers:
    
    Content-type - application/octet-stream
    x-ms-date - @{formatDateTime(utcnow(),'r')}
    Content-Length - 0
    x-ms-blob-type - BlockBlob
    

    enter image description here .

    Then call the action=flush and pass the position that you want the data to be flushed to e.g. here Position = no of characters in body you need to flush. e.g {Data_Count:2} - contains 14 characters

    URL - https://<storage_account_name>.dfs.core.windows.net/<container_name><folder_and_filepath>?action=flush&position=14&x-ms-version=2018-11-09
    
    Method - PATCH
    
    Body - {}
    
    Authentication- System-assigned managed identity
    
    Resource - https://storage.azure.com/
    
    Headers:
    
    x-ms-date - @{formatDateTime(utcnow(),'r')}
    

    enter image description here

    Output

    enter image description here