dvc

DVC - apply a pipeline to multiple source files

I have 100 source files to push through a pipeline. Is there a pattern for looping over a list, generating a new source and output file name each time for the same pipeline?

Thanks!

Solution

  • If you can elaborate more, that would be helpful, but maybe something like this is what you are looking for?

    dvc.yaml

    stages:
  example:
    foreach: ${some_param}
    do:
      cmd:
        - echo ${item}

    params.yaml

    some_param:
  - files/file1
  - files/file2
  - files/file3