pythonazureazure-data-lake-gen2

List and download files in a folder in a blob with blob level SAS token


I got a SAS token which was created for a specific folder on my Azure Datalake Gen2. The goal is, to download the folder with all its contents.

I understand, that I can create a DataLakeServiceClient, a FileSystemClient, or a DataLakeDirectoryClient as follows:

# configuration
url = 'https://my-account.blob.core.windows.net'
sas_token = '<sas-token>'
file_system_name =  'file_system_1'
subfolder_path = 'subfolder_1'

# service client
data_lake_service_client = DataLakeServiceClient(account_url=url, credential=sas_token)

# directory client and file system client
file_system_client = data_lake_service_client.get_file_system_client(file_system=file_system_name)
data_lake_directory_client = data_lake_service_client.get_directory_client(file_system=file_system_name, directory=subfolder_path)

Now to download specific files, I need to know what files exist:

How do I list and download all files in my directory?


Solution

  • I have reproduced in my environment and got expected results as below:

    In Portal created SAS for subfolder2 and below are files in that folder:

    enter image description here

    Code which worked for me :

    from azure.storage.filedatalake import DataLakeServiceClient
    import os
    
    url1 = "https://accountname.dfs.core.windows.net"
    sas = "sp=racwdlmeop&st=2023-08-30T10:46:22Z&se=2023-08-30T18:6:2Z&spr=https&sv=2022-11-02&sr=d&sig=kCkP25wyg%2A5"
    subfolderpath="folder2/subfolder2"
    data_lake_service_client = DataLakeServiceClient(account_url=url1, credential=sas)
    containername="rithwik"
    
    fsc = data_lake_service_client.get_file_system_client(file_system=containername)
    
    portalfolder = [path.name for path in fsc.get_paths(path=subfolderpath)]
    
    for fp in portalfolder:
        fct = fsc.get_file_client(file_path=fp)
        retrievedfile = fct.download_file()
        print(retrievedfile)
        localSaving = os.path.join(r'C:\Users\Desktop\New folder', fp)
        os.makedirs(os.path.dirname(localSaving), exist_ok=True)
        with open(localSaving, "wb") as f:
            f.write(retrievedfile.readall())
    

    Output:

    enter image description here

    Downloaded Locally:

    enter image description here