I have the following use case. I am working on a large project with multiple subtasks. For example, we are performing classification on some datasets, segmentation on others, and translation on yet another set of datasets (see below).
Each task has its own configuration. However, there is a part of the configuration that is shared among all tasks (such as the architecture of some network or specific blocks). This shared configuration is evolving, and each time I make a small change to it, I need to copy and paste the updated configuration into all the relevant folders.
Is there a way to centralize this configuration so that it can be imported into all the subtasks?
my_project
| classification_task
|--| conf
|--|--| config.yaml
|--|--| models
|--|--| datasets
|--| train.py
|--| ...
| segmentation_task
|--| conf
|--|--| config.yaml
|--|--| models
|--|--| datasets
|--| train.py
|--| ...
| translation_task
|--| conf
|--|--| config.yaml
|--|--| models
|--|--| datasets
|--| train.py
|--| ...
If I understand it correctly you have something like:
my_project
| classification_task
|--| conf
|--|--| config.yaml
|--|--| models
|--|--| datasets
| shared_config
|--| sharedconfig.yaml
You can use searchpath to look up for files listed in your defaults list that are stored somewhere else:
In your configs add:
hydra:
searchpath :
# relative option should also be possible if the structure is the same everywhere.
- file:///<global_path>/my_project/shared_config
defaults:
- sharedconfig
Maybe you can skip the searchpath and use ../../shared_config/sharedconfig in the defaults but I am not 100% sure.