python-xarrayzarrfsspec

getting KeyError '.zmetadata' when opening remote zarr store


Trying to read in a zarr store from s3 using xarray. Getting a Key Error. Any thoughts

import fsspec
import xarray as xr


uri = "s3://era5-pds/zarr/2020/12/data/eastward_wind_at_10_metres.zarr"
ds = xr.open_zarr(fsspec.get_mapper(uri, anon=True), consolidated=True)

I can open it locally ok if I download it first:

import s3fs
fs = s3fs.S3FileSystem(anon=True)
fs.get("s3://era5-pds/zarr/2020/12/data/eastward_wind_at_10_metres.zarr/*", "eastward_wind_at_10_metres.zarr", recursive=True)

Here's the Traceback associated with the top command

Traceback (most recent call last):
  File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/s3fs/core.py", line 234, in _call_s3
    return await method(**additional_kwargs)
  File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/aiobotocore/client.py", line 154, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.NoSuchKey: An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/fsspec/mapping.py", line 132, in __getitem__
    result = self.fs.cat(k)
  File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/fsspec/asyn.py", line 241, in cat
    raise ex
  File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/s3fs/core.py", line 737, in _cat_file
    resp = await self._call_s3(
  File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/s3fs/core.py", line 252, in _call_s3
    raise translate_boto_error(err) from err
FileNotFoundError: The specified key does not exist.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/xarray/backends/zarr.py", line 675, in open_zarr
    ds = open_dataset(
  File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/xarray/backends/api.py", line 572, in open_dataset
    store = opener(filename_or_obj, **extra_kwargs, **backend_kwargs)
  File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/xarray/backends/zarr.py", line 294, in open_group
    zarr_group = zarr.open_consolidated(store, **open_kwargs)
  File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/zarr/convenience.py", line 1178, in open_consolidated
    meta_store = ConsolidatedMetadataStore(store, metadata_key=metadata_key)
  File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/zarr/storage.py", line 2680, in __init__
    meta = json_loads(store[metadata_key])
  File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/fsspec/mapping.py", line 136, in __getitem__
    raise KeyError(key)
KeyError: '.zmetadata'

Solution

  • The problem is that this zarr dataset doesn't have consolidated metadata. The error is actually telling you this (KeyError: '.zmetadata').

    enter image description here