Azure Virtual Network does not allow access

I have two environments: staged and production. The following setup works in staged but not production. They are deployed using Arm templates so I can't understand why they won't work. I have the following setup:

App Service on subnet frontend with Network Security Group permitting access to internet
Blob storage on subnet backend
Virtual Network
Private endpoint for blob storage

On production when the app service tries to contact blob storage I get the following error. On staged I do not and everything works fine.

Azure.RequestFailedException: This request is not authorized to perform this operation.
RequestId:b648c22e-301e-0008-40e9-89a640000000
Time:2022-06-01T18:58:52.0008425Z
Status: 403 (This request is not authorized to perform this operation.)
ErrorCode: AuthorizationFailure

Content:
<?xml version="1.0" encoding="utf-8"?><Error><Code>AuthorizationFailure</Code><Message>This request is not authorized to perform this operation.
RequestId:b648d22e-301e-0008-40e9-75e640000000
Time:2022-06-01T19:58:52.0008425Z</Message></Error>

I have tried removing the private endpoint for blob storage in production but it has no effect. Only removing blob storage from the virtual network permits access. I have tried adding the reader and data access, data contributor, storage account contributor and owner roles to any managed identities that use the blog storage but that doesn't work either.

How can I find the differences between the two setups and make the blob storage work on the virtual network in production?

Solution

In case this helps anyone. The problem was with the DNS settings of the private endpoint. In Azure, if you go to Private Endpoint -> DNS Configuration it said something like this where the custom DNS record was missing (highlighted in red)

Whereas it should have shown the DNS record like so (it should not say 'to be configured correctly) but instead show a FQDN with an assigned IP address:

The section of my arm template that was missing was as follows:

    {
        "type": "Microsoft.Network/privateEndpoints/privateDnsZoneGroups",
        "apiVersion": "2020-11-01",
        "name": "[concat(variables('blobArchPrivateEndpointName'), '/default')]",
        "dependsOn": [
            "[resourceId('Microsoft.Network/privateEndpoints', variables('blobArchPrivateEndpointName'))]",
            "[resourceId('Microsoft.Network/privateDnsZones', variables('blobPrivateDnsName'))]"
        ],
        "properties": {
            "privateDnsZoneConfigs": [
                {
                    "name": "privatelink-blob-core-windows-net",
                    "properties": {
                        "privateDnsZoneId": "[resourceId('Microsoft.Network/privateDnsZones', variables('blobPrivateDnsName'))]"
                    }
                }
            ]
        }
    }

Once I included this, the access was given and problem solved