azureazure-databricksazure-rbac

com.databricks.sql.cloudfiles.errors.CloudFilesException: Failed to create an Event Grid subscription


I'm trying to use an autoloader to pick up files from a container in ADLS and store them as delta files in a delta table explicitly defined at an external location in a different container within the same storage account called "Bronze". When i try to write the stream with Trigger = Once option, i get an error " com.databricks.sql.cloudfiles.errors.CloudFilesException: Failed to create an Event Grid subscription.Please make sure that your service principal has 'write' permissions

The exact Error is as follows :

{ "error" : { "code" : "LinkedAuthorizationFailed", "message" : "The client '910adb4f-a9dd-4666-ace1-xxxxxxxxxx' with object id '910adb4f-a9dd-4666-ace1-xxxxxxxxxx' has permission to perform action 'Microsoft.EventGrid/eventSubscriptions/write' on scope '/subscriptions/485bf1a4-b8c5-48af-ba04-xxxxxxxxx/resourceGroups/StocksDev/providers/Microsoft.Storage/storageAccounts/stockpricesdev/providers/Microsoft.EventGrid/eventSubscriptions/databricks-query-8030d5e2-7058-4cfd-aa7f-b20867633d45-source-0'; however, it does not have permission to perform action(s) 'Microsoft.Storage/storageAccounts/write' on the linked scope(s) '/subscriptions/485bf1a4-b8c5-48af-ba04-xxxxxxxxxx/resourcegroups/StocksDev/providers/Microsoft.Storage/storageAccounts/stockpricesdev' (respectively) or the linked scope(s) are invalid." } }

I have now double checked permissions for the service principal and its according to docs:

  1. Storage blob data contributor on storage account - stockpricesdev

  2. Storage Queue Data contributor (I'm using the latest runtime > 8.0 without connection string so i guess i have added it just incase although not required)

  3. EventGrid EventSubscription Contributor on the resource group - StockDev

I can also see the inherited role assignments within scopes. I'm not sure why this is happening as a contributor role would be sufficient to perform write action


Solution

  • The error occurred as the roles you are currently using (Storage Blob Data Contributor) does not have Microsoft.Storage/storageAccounts/write permission, which is required to create Event Grid subscription.

    To resolve the error, you need to assign either Contributor or Storage Account Contributor role to the service principal under your storage account.

    Go to Azure Portal -> Storage Accounts -> Select your Storage account -> Access control (IAM) -> Add role assignment -> Contributor

    enter image description here

    For your storage account, you require:

    Within the resource group:

    Reference: Required permissions for configuring file notification - Azure Databricks | Microsoft