I am using hydra for configuration management.
When my configuration files contain parts of paths (folder names, file names), what is the most convenient way to create a full path to a given file? As an example:
I have a config.yaml that contains some folder names:
paths:
data:
base: data # top-level data folder in my project
external: external # data as loaded from other sources
raw: raw # data as in external, but stored as binaries
dataset:
My configuration-files in dataset contain file names, for example
files:
train: train_data
test: test_data
Finally, I have defined my own subclasses of DictConfig for better type hinting in my editor.
What I would like:
>>> cfg.dataset.files.train
Path("base/raw/train_data")
But currently, I always have to write
>>> Path(cfg.paths.data.base)/cfg.paths.data.raw/cfg.dataset.files.train
Path("base/raw/train_data")
Is there a hydra-based solution? Or should I rather create the full filepaths in my `DictConfig` subclasses? Or should I completely abandon storing filepaths in configs?
You can use OmegaConf interpolations:
Note that those are relative (see the prefix ., without it the interpolations would point to an absolute path in the config object).
paths:
data:
base: data
external: ${.base}/external # evaluates at access to data/external
raw: ${.base}/raw