azureazure-blob-storageazure-data-factoryazure-storage-queues

Getting Azure Blob Queues Message from Azure Data Factory Web Activity


I am trying to use Azure Web Activity to get the queues message from Azure Blob Storage. The message will then be parsed to obtain the parameter that I need in subsequent part of the data pipeline.

enter image description here

I am attempting to connect using SAS but keep getting

<?xml version="1.0" encoding="utf-8"?><Error><Code>UnsupportedHttpVerb</Code><Message>The resource doesn't support specified Http Verb.

What is the problem with the my setting here? I have read through Azure Documentation but cannot seem to find any solution. Thank you.


Solution

  • I tried with below URL using SAS and got the expected result.

    https://<storage_account_name>.queue.core.windows.net/<message_queue_name>/messages<SAS>
    

    This is my Pipeline JSON:

    {
        "name": "pipeline2",
        "properties": {
            "activities": [
                {
                    "name": "Web1",
                    "type": "WebActivity",
                    "dependsOn": [],
                    "policy": {
                        "timeout": "0.12:00:00",
                        "retry": 0,
                        "retryIntervalInSeconds": 30,
                        "secureOutput": false,
                        "secureInput": false
                    },
                    "userProperties": [],
                    "typeProperties": {
                        "url": "https://<storage_account_name>.queue.core.windows.net/<message_queue_name>/messages<SAS>",
                        "method": "GET",
                        "headers": {
                            "x-ms-date": {
                                "value": "@{formatDateTime(utcnow(),'r')}",
                                "type": "Expression"
                            },
                            "x-ms-version": "2020-04-08",
                            "Content-Type": "application/json"
                        }
                    }
                }
            ],
            "annotations": []
        }
    }
    

    If you check the below checkmark in the queue, you will get the Encrypted message body in the web activity output.

    enter image description here

    Output:

    enter image description here

    Uncheck it and you can see, I got the message body in the web activity output.

    enter image description here