pythonazureazure-blob-storagezarr

Upload zarr folder to Azure blob container

I tried to upload a zarr file (folder-like) to an Azure container using Python but it does not work properly, as it only uploaded the innermost files and deleted everything else in the container. This is my code:

def upload_zarr(file_path):
    credential = env_vars['STORAGE_ACCOUNT_KEY']
    container_client = ContainerClient(account_url='https://<account-name>.blob.core.windows.net',
                                      container_name='data',
                                      credential=credential)
    store = zarr.ABSStore(client=container_client, prefix='/')
    ds = zarr.open(file_path)
    zarr.save(store, ds)
    print("finished writing to Azure")

I don't think recursively upload the whole file as a folder recursively works since it erases the .zarr properties.

Solution

  • Upload zarr folder to Azure blob container

    You can use the zarr.convenience.copy() method to Upload the zarr file to the Azure blob storage.

    Code:

    import zarr
from azure.storage.blob import ContainerClient 

def upload_zarr(file_path):
    credential = env_vars['STORAGE_ACCOUNT_KEY']
    container_client = ContainerClient(account_url='https://<account-name>.blob.core.windows.net',
                                      container_name='test',
                                      credential=credential)
    store = zarr.ABSStore(client=container_client, prefix='/')
    ds = zarr.open(file_path)
    group = zarr.group(store=store)
    zarr.convenience.copy(ds, group, 'sample.zarr')
print("finished writing to Azure")

    Output:

    finished writing to Azure

    enter image description here

    enter image description here

    Reference:

    Convenience functions (zarr.convenience) — zarr 2.16.1 documentation