Using multiple configs in the same group to interpolate values in a yaml file

In Hydra I have the following configuration:

├── conf
    │   ├── config.yaml
    │   ├── callbacks
    │   │   ├── callback_01.yaml
    │   │   └── callback_02.yaml
    │   └── trainer
    │       ├── default.yaml

The callbacks have a structure like this:

_target_: callback_to_instantiate

I need to pass to the trainer/default.yaml both the callbacks through interpolation. I tried like this:

    _target_: pytorch_lightning.Trainer
    callbacks:
        - ${callbacks.callback_01}
        - ${callbacks.callback_02}

With the config.yaml like this:

    defaults:
      - _self_
      - trainer: default

I did also other trials but it doesn't seem to work. Is there a way to interpolate like that in a yaml file by using two or more yaml files that are in the config group? I would like if possible to keep this structure.

Solution

Currently the recommended approach is:

compose a mapping whose values are the desired callbacks, and then
use the oc.dict.values OmegaConf resolver to get a list of values from that dictionary.

# conf/config.yaml
defaults:
  - callbacks@_callback_dict.cb1: callback_01
  - callbacks@_callback_dict.cb2: callback_02
  - trainer: default
  - _self_

# conf/trainer/default.yaml
_target_: pytorch_lightning.Trainer
callbacks: ${oc.dict.values:_callback_dict}

# my_app.py
from typing import Any
import hydra
from omegaconf import DictConfig, OmegaConf

@hydra.main(config_path="conf", config_name="config")
def app(cfg: DictConfig) -> Any:
    OmegaConf.resolve(cfg)
    del cfg._callback_dict
    print(OmegaConf.to_yaml(cfg))

if __name__ == "__main__":
    app()

At the command line:

$ python my_app.py
trainer:
  _target_: pytorch_lightning.Trainer
  callbacks:
  - _target_: callback_to_instantiate_01
  - _target_: callback_to_instantiate_02

For reference, there is an open issue on Hydra's github repo advocating for an improved user experience around