databricksdelta-live-tables

Different storage paths depending on checkout branch with Delta Live Tables


How can I change the storage location depending on what branch I am working on. For example, I'd like the storage location when running a DLT pipeline on my feature branch to be different from the storage location when running the pipeline on a development branch.


Solution

  • For that, you need to create a separate DLT pipeline per branch. Each branch will have different DLT pipeline settings that will point to different schemas and locations - that approach is recommended by the DLT docs.

    Following repo has an example of doing that with Terraform. Or you can use Databricks Asset Bundles - it may also simplify the promotion of code between branches/environments.