azureazure-storageazure-data-lake-gen2

list files and folders inside an azure DataLake storage Gen2 container using SAS at a directory level


I have a storage account which has the hierarchical namespace feature enabled (data lake storage gen 2) with the following file structure.

container/folder1/folder2/folder3/file1.txt container/folder1/folder2/folder3/file2.txt

I created a SAS token on the directory 2 level. Here is my generated URI

https://testtest123test.blob.core.windows.net/container/folder1/folder2/?sv=2024-11-04&se=2024-10-04T21%3A44%3A27Z&sr=d&sp=rl&sdd=2&sig=removed.

I want to list all the files and folders inside Folder 2.

I added "&restype=directory&comp=list" at the end of my URL ,but I got the following error :

<?xml version="1.0" encoding="utf-8"?>
<Error>
    <Code>InvalidQueryParameterValue</Code>
    <Message>Value for one of the query parameters specified in the request URI is invalid.
RequestId:3ba2501f-d01e-0012-15a6-16e717000000
Time:2024-10-04T21:43:25.7972913Z</Message>
    <QueryParameterName>comp</QueryParameterName>
    <QueryParameterValue>list</QueryParameterValue>
</Error>

I also tried adding "&restype=container&comp=list" at the end of my URL, but I got the following error :

<?xml version="1.0" encoding="utf-8"?>
<Error>
    <Code>InvalidUri</Code>
    <Message>The requested URI does not represent any resource on the server.
RequestId:128609cc-b01e-0059-69a6-161b44000000
Time:2024-10-04T21:45:53.6788261Z</Message>
</Error>

Can anyone help?

thanks


Solution

  • list files and folders inside an azure DataLake storage Gen2 container using SAS at a directory level.

    You can use the below provided Rest API to list the second level of folder and files in Azure Data lake Storage Gen2.

    In my environment, I created SAS token from directory level(folder2)through portal with rl(read and list) permission.

    Portal: enter image description here

    Request:

    https://<storage account name>.dfs.core.windows.net/<Conatiner name>?directory=folder1/folder2&recursive=true&resource=filesystem&sp=rl&st=2024-10-07T03:52:29Z&se=2024-10-07T11:52:29Z&spr=https&sv=2022-11-02&sr=d&sig=removed&sdd=2
    

    Output:

    {
      "paths": [
        {
          "contentLength": "5027",
          "creationTime": "1xxxx548",
          "etag": "0xxxxxFDC",
          "expiryTime": "0",
          "group": "$superuser",
          "lastModified": "Mon, 07 Oct 2024 03:42:01 GMT",
          "name": "folder1/folder2/01-10-2024.html",
          "owner": "$superuser",
          "permissions": "rw-r-----"
        },
        {
          "contentLength": "9463",
          "creationTime": "13xxxx528",
          "etag": "0xxF2FC8",xx
          "expiryTime": "0",
          "group": "$superuser",
          "lastModified": "Mon, 07 Oct 2024 03:42:01 GMT",
          "name": "folder1/folder2/04-10-2024.html",
          "owner": "$superuser",
          "permissions": "rw-r-----"
        },
        {
          "contentLength": "0",
          "creationTime": "13xxx1444",
          "etag": "0xx851464",xxx
          "expiryTime": "0",
          "group": "$superuser",
          "isDirectory": "true",
          "lastModified": "Mon, 07 Oct 2024 03:41:15 GMT",
          "name": "folder1/folder2/folder3",
          "owner": "$superuser",
          "permissions": "rwxr-x---"
        },
        {
          "contentLength": "5027",
          "creationTime": "13xxxx123264",
          "etag": "0x8xxxC6840",
          "expiryTime": "0",
          "group": "$superuser",
          "lastModified": "Mon, 07 Oct 2024 03:41:52 GMT",
          "name": "folder1/folder2/folder3/01-10-2024.html",
          "owner": "$superuser",
          "permissions": "rw-r-----"
        },
        {
          "contentLength": "9463",
          "creationTime": "13xxx3244",
          "etag": "0xxxxxC682C",
          "expiryTime": "0",
          "group": "$superuser",
          "lastModified": "Mon, 07 Oct 2024 03:41:52 GMT",
          "name": "folder1/folder2/folder3/04-10-2024.html",
          "owner": "$superuser",
          "permissions": "rw-r-----"
        }
      ]
    }
    

    enter image description here

    Reference: Path - List - REST API (Azure Storage Services) | Microsoft Learn