pythongitfb-hydraomegaconf

Make Hydra to load configs from zip-archive

I am trying to adopt this wonderful repository with Lightning-Hydra template to my workflow.

My workflow looks like following.

  1. Create a new git branch for a new experiment.

  2. Write a code (plus configs) to implement an idea (including debugging and testing)

  3. Commit

  4. Pack everything into a zip archive (zip -rp experiment-$(git rev-parse HEAD).zip *)

  5. Add to the archive the file __main__.py, calling main()

    This file helps python interpreter to recognize the archive as a python script and allows the call python experiment-aabbcc.zip

    Approximate content of this file is

    import sys
from src.train import main
main(sys.argv[1:])

  6. Submit to the job queue command python /path/to/experiment-aabbcc.zip --other --command --line-args.

    I use this fork of the task-spooler as a job scheduler. Because of technical limitations of the PC that I use, I cannot run docker (actually I ssh to a docker container that doesn't support docker-in-docker, and I cannot change that).

  7. git checkout master

  8. go to 1

The issue is that hydra + OmegaConf refuse to load configs from created zip archive, they always look into a file system.

Since I frequently switch branches, content of working directory often changes, and I have high chances that a job submitted to a job queue will run the code that is different from what was intended.

So, I am looking for a solution that would allow me to isolate a code, submitted to the task spooler, without making extra copies of a repo on a file system.

Solution

  • In principle this potentially be implemented using a ConfigSource plugin. Example here.