Azure Web App Cannot Access Azure Container Registry Using Managed Identity

I'm trying to deploy a multi-container setup on Azure App Service using images stored in an Azure Container Registry (ACR). Both my App Service and ACR are configured to use managed identity, and I have assigned the ACR Pull role to the App Service's managed identity.

I have been able to connect to the registry using Azure CLI and I've been able to push images from my pipeline.

However, when the App Service tries to pull the images, I encounter the following error in the logs:

DockerApiException: Docker API responded with status code=InternalServerError, response={"message":"Head \"https://<my-registry>.azurecr.io/v2/<image>/manifests/latest\": unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information."}

My Setup:

App Service Managed Identity: Enabled and set to "On". Assigned the ACR Pull role at the scope of the ACR.

ACR Managed Identity: Enabled and set to "On".

Deployment Pipeline: I build and push the Docker images to ACR via an Azure DevOps pipeline. Here is an excerpt from my pipeline configuration:

trigger:
- main

variables:
  buildConfiguration: 'Release'
  appServiceName: 'myAppService'
  backendImageName: 'myregistry.azurecr.io/my-backend'
  frontendImageName: 'myregistry.azurecr.io/my-frontend'
  resourceGroup: 'myResourceGroup'
  subscriptionId: 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'

stages:
# Stage 1: Build and Push Docker Images
- stage: BuildAndPush
  displayName: Build and Push Docker Images to Azure Container Registry
  jobs:
  - job: BuildAndPush
    displayName: Build and Push Backend and Frontend Docker Images
    pool:
      vmImage: 'ubuntu-latest' # Use Ubuntu for Docker tasks
    steps:
    # Step 1: Login to Azure Container Registry
    - task: Docker@2
      displayName: Login to Azure Container Registry
      inputs:
        command: login
        containerRegistry: 'my_container_registry'

    # Step 2: Build and Push Backend Docker Image
    - task: Docker@2
      displayName: Build and Push Backend Docker Image
      inputs:
        command: buildAndPush
        dockerfile: ./Dockerfile
        context: ./Backend.Api
        repository: my-backend
        containerRegistry: 'my_container_registry'
        tags: |
          latest
          $(Build.BuildId)

    # Step 3: Build and Push Frontend Docker Image
    - task: Docker@2
      displayName: Build and Push Frontend Docker Image
      inputs:
        command: buildAndPush
        dockerfile: ./Frontend.Client/Dockerfile
        context: ./Frontend.Client
        repository: my-frontend
        containerRegistry: 'my_container_registry'
        tags: |
          latest
          $(Build.BuildId)

# Stage 2: Deploy to Azure
- stage: Deploy
  displayName: Deploy Docker Containers to Azure App Service
  dependsOn: BuildAndPush
  condition: succeeded()
  jobs:
  - job: DeployToAzure
    displayName: Deploy Containers to Azure App Service
    pool:
      vmImage: 'ubuntu-latest'
    steps:
    # Step 1: Login to Azure CLI
    - task: AzureCLI@2
      displayName: Login to Azure
      inputs:
        azureSubscription: 'Azure subscription 1(xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)'
        scriptType: bash
        scriptLocation: inlineScript
        inlineScript: |
          echo "Logging in to Azure CLI"
    - task: Docker@2
      displayName: Login to Azure Container Registry
      inputs:
        command: login
        containerRegistry: 'my_container_registry'

    # Step 2: Generate Docker Compose File
    - script: |
        echo "
        version: '3.8'
        services:
          backend:
            image: $(backendImageName):latest
            ports:
              - 5000:80
          frontend:
            image: $(frontendImageName):latest
            ports:
              - 3000:80
        " > docker-compose.yml
      displayName: Generate Docker Compose File

    # Step 3: Configure App Service for Multi-Container Deployment
    - task: AzureCLI@2
      displayName: Configure Azure App Service with Docker Compose
      inputs:
        azureSubscription: 'Azure subscription 1(xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)'
        scriptType: bash
        scriptLocation: inlineScript
        inlineScript: |
          az webapp config container set \
            --name $(appServiceName) \
            --resource-group $(resourceGroup) \
            --subscription $(subscriptionId) \
            --multicontainer-config-type compose \
            --multicontainer-config-file docker-compose.yml

    # Step 4: Restart App Service to Apply Changes
    - task: AzureCLI@2
      displayName: Restart Azure App Service
      inputs:
        azureSubscription: 'Azure subscription 1(xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)'
        scriptType: bash
        scriptLocation: inlineScript
        inlineScript: |
          az webapp restart \
            --name $(appServiceName) \
            --resource-group $(resourceGroup) \
            --subscription $(subscriptionId)

In the final step, I configure my App Service using this docker-compose.yml file:

version: '3.8'
services:
  backend:
    image: <my-registry>.azurecr.io/backend:latest
    ports:
      - 5000:80
  frontend:
    image: <my-registry>.azurecr.io/frontend:latest
    ports:
      - 3000:80

I then restart the App Service to apply the changes.

Troubleshooting Steps So Far:

1. Verified that the managed identity is enabled for both the App Service and the ACR.
2. Checked that the ACR Pull role is assigned to the App Service's managed identity.
3. Confirmed that the Docker images exist in the ACR and can be pulled locally using the same docker-compose.yml.
4. Checked for network restrictions or firewalls that might block the connection between the App Service and the ACR.

Despite these steps, the App Service still cannot pull the images from the ACR. The full logs are attached for additional context.

Logs Excerpt: DockerApiException: Docker API responded with status code=InternalServerError, response={"message":"Head \"https://<my-registry>.azurecr.io/v2/<image>/manifests/latest\": unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information."}

Questions:

1. What additional configurations are required to allow the App Service to pull images from ACR using managed identity?
2. Are there specific network security group (NSG) settings I need to verify?

Any help or guidance would be greatly appreciated!

I expected the Azure App Service to successfully pull the Docker images from the Azure Container Registry using the managed identity. Specifically, I anticipated that:

1. The App Service would use its system-assigned managed identity to authenticate with ACR.
2. The ACR Pull role would provide sufficient permissions for pulling the images.
3. The containers would start successfully without any authentication or authorization issues.

I encountered the same issue and was able to resolve it by ensuring the Web App is configured to use its managed identity for ACR authentication. Here’s what I did:

Assign the AcrPull Role: Ensure the Web App's system-assigned managed identity has the AcrPull role assigned to the Azure Container Registry. You can do this using the following Azure CLI command:

az role assignment create --assignee <managed-identity-principal-id> --role AcrPull --scope /subscriptions/<subscription-id>/resourceGroups/<resource-group-name>/providers/Microsoft.ContainerRegistry/registries/<acr-name>

Enable Managed Identity Authentication: Update the Web App’s configuration to use its managed identity for ACR authentication.

Run this command:

az resource update --ids /subscriptions/<subscription-id>/resourceGroups/<resource-group-name>/providers/Microsoft.Web/sites/<app-name>/config/web --set properties.acrUseManagedIdentityCreds=True

Restart the Web App: Finally, restart the Web App to apply the changes:

az webapp restart --name <app-name> --resource-group <resource-group-name>

After performing these steps, the Web App was able to successfully pull the container image from the ACR without any issues.