python-xarrayopendappython-siphon

read in authorized opendap url using xarray


I would like to open a opendap url using xarray. It requires authorization as it held at UCAR RDA:

https://rda.ucar.edu/datasets/ds084.1/#!description

The url for one file is as follows 'https://rda.ucar.edu/thredds/dodsC/files/g/ds084.1/2020/20200101/gfs.0p25.2020010100.f000.grib2'

I'm not sure if I can pass in the authorization as a backend_kwarg?

The below code will give an error message

import xarray as xr
url = "https://rda.ucar.edu/thredds/dodsC/files/g/ds084.1/2020/20200101/gfs.0p25.2020010100.f000.grib2"
ds = xr.open_dataset(url)
Traceback (most recent call last):
  File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/xarray/backends/file_manager.py", line 199, in _acquire_with_cache_info
    file = self._cache[self._key]
  File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/xarray/backends/lru_cache.py", line 53, in __getitem__
    value = self._cache[key]
KeyError: [<class 'netCDF4._netCDF4.Dataset'>, ('https://rda.ucar.edu/thredds/dodsC/files/g/ds084.1/2020/20200101/gfs.0p25.2020010100.f000.grib2',), 'r', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False))]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/xarray/backends/api.py", line 572, in open_dataset
    store = opener(filename_or_obj, **extra_kwargs, **backend_kwargs)
  File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/xarray/backends/netCDF4_.py", line 364, in open
    return cls(manager, group=group, mode=mode, lock=lock, autoclose=autoclose)
  File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/xarray/backends/netCDF4_.py", line 314, in __init__
    self.format = self.ds.data_model
  File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/xarray/backends/netCDF4_.py", line 373, in ds
    return self._acquire()
  File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/xarray/backends/netCDF4_.py", line 367, in _acquire
    with self._manager.acquire_context(needs_lock) as root:
  File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/xarray/backends/file_manager.py", line 187, in acquire_context
    file, cached = self._acquire_with_cache_info(needs_lock)
  File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/xarray/backends/file_manager.py", line 205, in _acquire_with_cache_info
    file = self._opener(*self._args, **kwargs)
  File "netCDF4/_netCDF4.pyx", line 2357, in netCDF4._netCDF4.Dataset.__init__
  File "netCDF4/_netCDF4.pyx", line 1925, in netCDF4._netCDF4._ensure_nc_success
OSError: [Errno -78] NetCDF: Authorization failure: b'https://rda.ucar.edu/thredds/dodsC/files/g/ds084.1/2020/20200101/gfs.0p25.2020010100.f000.grib2'

Siphon's session_manager may hint at what auth looks like https://unidata.github.io/siphon/latest/examples/Basic_Usage.html#sphx-glr-examples-basic-usage-py / https://github.com/Unidata/siphon/blob/master/siphon/http_util.py#L52


Solution

  • Thanks to Ryan May for pointing me to https://publicwiki.deltares.nl/display/OET/Accessing+netCDF+data+via+OPeNDAP+on+password+protected+servers

    Creating the dot file in my home dir allowed me to read in the url. May not be the cleanest and I imagine can cause issues on VM's/cluster but works. Still hoping for a backend_kwargs approach.

    Create a file .netrc in your home directory which looks like this:

    machine rda.ucar.edu
    login USR
    password PWD
    

    and a file .dodsrc in you home directory which look like this:

    HTTP.COOKIEJAR=<HOME_DIR>/.cookies
    HTTP.NETRC=<HOME_DIR>/.netrc
    

    You can now pass urls which require authentication:

    import xarray as xr
    url = "https://rda.ucar.edu/thredds/dodsC/files/g/ds084.1/2020/20200101/gfs.0p25.2020010100.f000.grib2"
    ds = xr.open_dataset(url)