pythoncollectionscommand-line-interfaceomegaconf

How to provide a List/Tuple/Collection to OmegaConf's structured config from Command Line Interface (CLI)?

I am using OmegaConf's structured config system in order to get a convenient, strongly typed way of configuring my Python application, since it takes very few lines of code and allows merging confs from source, filesystem and CLI. This is defined through a ConfDef class like the one below.

Now, I want to provide a parameter as a collection of, say, strings. While the OmegaConf documentation addresses how should the ConfDef look like, no details about the CLI interaction are provided. Hence the question:

How can we provide a collection of values when calling the Python app from CLI using structured OmegaConf?

I am using Ubuntu20.04, but generic approaches if existing would be preferred.

from dataclasses import dataclass
from typing import List
from omegaconf import OmegaConf, MISSING


@dataclass
class ConfDef:
    """
    :cvar MY_PATHS: Collection of paths to be processed
    """
    MY_PATHS: List[str] = MISSING
Solution

  • In Linux bash, we are expected to provide the collection in the form:

    python my_scripy.py MY_PATHS="[a,b,c]"

    Then, inside Python, we can instantiate and access the conf e.g. as follows:

    conf = OmegaConf.structured(ConfDef())
cli_conf = OmegaConf.from_cli()
conf = OmegaConf.merge(conf, cli_conf)
print("\n\nCONFIGURATION:")
print(OmegaConf.to_yaml(conf), end="\n\n\n")

print(conf.MY_PATHS)  # This prints the list of strings ["a", "b", "c"]

    Furthermore, if you want to expand a glob pattern with multiple paths, e.g. ./data/files/*, this can be achieved as follows:

    # just basenames
python my_script.py MY_PATHS="[`ls -p data/files/ | grep -v / | tr '\n' ','`]"
# absolute paths (nonrecursive)
python my_script.py MY_PATHS="[`EXCEL_PATHS="[`ls -d $PWD/data/files/* | tr '\n' ','`]"`]"

    Explanation (from here):

    Yielding the collection in the desired form above.