databricksdatabricks-asset-bundle

Override the libraries settings for all tasks of a specific target in databricks asset bundles


How can I override the task settings for all tasks in a specific target without having to name each task by its task_key individually?

I know I can do the following to override the libraries setting for specific tasks in my dev target. But I need to (repetitively) list every single task in my job, which can get quite long.

Is there a way to override the libraries setting for ALL tasks in my target? The wildcard operator for task_key does not work / does not seem to be implemented.

targets:
  dev:
    mode: development
    workspace:
      host: https://xxx-xxxxxx.x.azuredatabricks.net
    resources:
      jobs:
        my_job_name:
          tasks:
            - task_key: ingestion
              libraries:
                - whl: ./dist/*.whl
                - pypi:
                    package: some-package==${var.some_package_version}
                    repo: https://pkgs.dev.azure.com/stihl/xxxx/_packaging/xxx/pypi/simple/
            - task_key: preparation
              libraries:
                - whl: ./dist/*.whl
                - pypi:
                    package: some-package==${var.some_package_version}
                    repo: https://pkgs.dev.azure.com/stihl/xxxx/_packaging/xxx/pypi/simple/

Solution

  • I was able to achieve this with yaml anchoring. It feels a bit jenky, but seems to work and cuts down on that boilerplate.

    I will leave only the important parts of the yaml in the answer here. The important part is the libs property, and how its referenced under the task_key.

    ---
    # deployment.yml
    ---
    resources:
      jobs:
        example-dab-workflow:
          libs: &libs
            libraries:
              - pypi:
                  package: llama-index==0.10.43
              - pypi:
                  package: pytest==8.3.2
              - pypi:
                  package: transformers==4.41.1
              - whl: ../dist/*.whl
    
          name:  "example-dab"
    
          tasks:
            - task_key: example_task
              existing_cluster_id: abc1234-foo999-barre
              python_wheel_task:
                package_name: example_dab
                entry_point: "example_entrypoint"
              <<: *libs