databricksdatabricks-clidatabricks-asset-bundle

databricks asset bundle clusters for dev and prod

I am using Databricks bundles I have a dev and prod environment. I have a yaml that looks something like this:

# yaml-language-server: $schema=bundle_config_schema.json
bundle:
  name: baby-names

resources:
      tasks:
        - task_key: retrieve-baby-names-task
          existint_cluster: 1234
          notebook_task:
            notebook_path: ./retrieve-baby-names.py

targets:
  development:
    workspace:
      host: <workspace-url>
  production:
    workspace:
      host: <workspace-url>

This works great if you have the same cluster id in multiple environments, I don't I see Jinja is not supported. How can I set some logic that lets me deploy to env A with cluster id related to that env? vs b and its cluster id. this seems fundamental.

I have tried manually copy and pasting the new ID's which isn't what I want to do.

Solution

  • The best solution I found was to use the Jinja package in python and in my build tool have a task that dynamically creates the yaml with the values for that environment.