pythonfb-hydra

How to specify parent directory of outputs directory in hydra?


When using hydra to manage hyperparameter searches, by default, the output directory is saved at the current working directory / hydra.core.hydra_config.HydraConfig.get().run.dir. Is there a way to change that behavior to save the info at some_folder_I_specify/ hydra.core.hydra_config.HydraConfig.get().run.dir instead? Ideally, I would specify the location of the output folder via the command line.

The situation is, that I have my directory containing my repo with the script, another folder for all the generated data and I want to execute the code in a third folder. By default, the outputs will be saved in the third folder, while I'd like to save them in the data folder.

There are a number of tutorials on how to change the second part of the output path, starting from the cwd and how to change the cwd to the output folder, but I cannot find anything an changing the first part.


Solution

  • You can customize working directory pattern.

    For sweep, you can override hydra.sweep.dir and hydra.sweep.subdir from the command line or from your config files. In general, multiple runs of the same sweep will be written into the same parent directory (specificed by hydra.sweep.dir).

    Having them written a a different parent directory is not formally supported, but a hack can be to set hydra.sweep.dir to the common root or even the disk root (e.g. /) and have the entire directory specified in hydra.sweep.subdir. This is not supported because it might cause some per sweep files to be written to weird locations and also to be overwritten but different sweeps.