Using distcp to copy to Azure ADLS Gen1 fails with 403

I am trying to copy to Azure Data Lake Storage (ADLS) Gen1, while authenticating using OAuth2.

I am getting the following error:

com.microsoft.azure.datalake.store.ADLException: Error getting info for file /myContainer Operation GETFILESTATUS failed with HTTP403 : null

Here's how my distcp looks like

hadoop distcp 
    -Dfs.adl.oauth2.access.token.provider.type=ClientCredential 
    -Dfs.adl.oauth2.client.id=<client_id>
    -Dfs.adl.oauth2.credential=<key>
    -Dfs.adl.oauth2.refresh.url=https://login.microsoftonline.com/*****/oauth2/token 
hdfs:///path/to/file 
adl://adlsgen1.blob.core.windows.net/myContainer

Any idea what could cause this?

If you want to use Azure AD to access Azure data lake gen2 with a service principal, we need to configure RABC role for the service principal.

For example

1. Create a service principal and assign Storage Blob Data Owner to the sp.(I use Azure CLI).

az ad sp create-for-rbac -n "MyApp" --role "Storage Blob Data Owner"\
    --scopes /subscriptions/{SubID}/resourceGroups/{ResourceGroup1} \
    /subscriptions/{SubID}/resourceGroups/{ResourceGroup2}

2. configure

hadoop distcp 
    -D fs.azure.account.auth.type=OAuth 
    -D fs.azure.account.oauth.provider.type=org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider
    -D fs.azure.account.oauth2.client.endpoint=[token endpoint],
    -D fs.azure.account.oauth2.client.id=[Application client ID],
    -D fs.azure.account.oauth2.client.secret=[client secret]
hdfs:///path/to/file 
abfs://container1@abfswales1.dfs.core.windows.net/

For more details, please refer to the document and the document