Creating an Azure Batch Service Pool using an ADF pipeline

I want to automatically create an Azure Batch Service pool when an ADF pipeline is run.

I created a web activity inside the pipeline and passed the parameters under settings as follows:

URL: https://management.azure.com/subscriptions/{Subscription ID}/resourceGroups/{Resource Group Name}/providers/Microsoft.Batch/batchAccounts/{Azure Batch Account Name}/pools?api-version=2023-05-01 Method: POST Body: { "id": "pool2", "vmSize": "standard_a1", "virtualMachineConfiguration": { "imageReference": { "publisher": "Canonical", "offer": "0001-com-ubuntu-server-focal", "sku": "20_04-lts", "version": "latest" }, "nodeAgentSKUId": "batch.node.ubuntu 20.04" }, "resizeTimeout": "PT15M", "targetDedicatedNodes": 5, "targetLowPriorityNodes": 0, "taskSlotsPerNode": 3, "taskSchedulingPolicy": { "nodeFillType": "spread" }, "enableAutoScale": false, "enableInterNodeCommunication": true, "metadata": [ { "name": "myproperty", "value": "myvalue" } ] } as per the following official documentation (https://learn.microsoft.com/en-us/rest/api/batchservice/pool/add?tabs=HTTP) Authentication: None Resource: https://learn.microsoft.com/en-us/rest/api/batchservice/pool/add? Headers: Content-Type --> application/json, Authorization --> Basic @{base64(concat('{Batch Account Name}', ':', '{Batch Account Primary Key}'))}

But I keep getting the following error:

These are the following Authentication types that are currently being displayed under settings:

I also tried removing the Authorization header, but I am getting the following error:

I even tried changing the Authentication to 'System Assigned Managed Identity'. However, ADF says that resource doesn't exist which is weird because the same works for GET and DELETE methods respectively.

So not exactly sure how to provide the authorization header in the correct format or how to generate a refresh token for Azure Batch Account. Any help would be greatly appreciated. Please let me know if something is not clear.

Solved the issue by creating another web activity for generating the bearer token whose settings are as follows:

URL: https://login.microsoftonline.com/{tenant-id}/oauth2/token

Method: POST

Body: grant_type=client_credentials&client_id={client-id}@{tenant-id}&client_secret={client-secret}&resource={domain-name}/@{tenant-id}

Headers: Concent-Type --> application/x-www-form-urlencoded

The access token is then passed as input to the next web activity whose settings are as follows:

URL: https://management.azure.com/subscriptions/{subscription-id}/resourceGroups/{resource-group-name}/providers/Microsoft.Batch/batchAccounts/{batch-account-name}/pools/{give the name of the test pool that you are trying to create}?api-version=2023-05-01

Method: PUT

Body: { "id": "/subscriptions/{subscription-id}/resourceGroups/resource-group-name/providers/Microsoft.Batch/batchAccounts/{batch-account-name}/pools/{name of the pool that is about to be created}", "name": "testpool", "type": "Microsoft.Batch/batchAccounts/pools", "etag": "W/"0x8D4EDFEBFADF4AB"", "properties": { "lastModified": "2017-08-28T10:22:55.9407275Z", "creationTime": "2017-08-28T10:22:55.9407275Z", "provisioningState": "Succeeded", "provisioningStateTransitionTime": "2017-08-28T10:22:55.9407275Z", "allocationState": "Resizing", "allocationStateTransitionTime": "2017-08-28T10:22:55.9407275Z", "vmSize": "STANDARD_A2", "interNodeCommunication": "Disabled", "taskSlotsPerNode": 1, "taskSchedulingPolicy": { "nodeFillType": "Spread" }, "deploymentConfiguration": { "virtualMachineConfiguration": { "imageReference": { "publisher": "Canonical", "offer": "UbuntuServer", "sku": "18.04-LTS", "version": "latest" }, "nodeAgentSkuId": "batch.node.ubuntu 18.04" } }, "scaleSettings": { "autoScale": { "formula": "$TargetDedicatedNodes=1", "evaluationInterval": "PT5M" } }, "currentDedicatedNodes": 0, "currentLowPriorityNodes": 0 } }

This JSON script was extracted from https://learn.microsoft.com/en-us/rest/api/batchmanagement/pool/create?tabs=HTTP

Authentication: None

Headers: Content-Type --> application/json; Authorization --> @{concat('Bearer ',activity('Generate bearer token').output.access_token)}

The client-id and client-secret can be generated by registering a new app in the Azure portal and linking it to the Batch Account. The microsoft official documentation for doing this can be found at: https://learn.microsoft.com/en-us/azure/batch/batch-aad-auth. Just follow all the steps until "Assign Azure RBAC to your application" and you are good to go.

Once all the steps are implemented successfully, you can find the new pool under the Azure batch account.

Please note that if you can create a pool, but cannot use it, then you have to reach out to the Microsoft support team by requesting a quota increase as follows: