How can I override the task settings for all tasks in a specific target without having to name each task by its task_key individually?
I know I can do the following to override the libraries
setting for specific tasks in my dev
target. But I need to (repetitively) list every single task in my job, which can get quite long.
Is there a way to override the libraries
setting for ALL tasks in my target? The wildcard operator for task_key
does not work / does not seem to be implemented.
targets:
dev:
mode: development
workspace:
host: https://xxx-xxxxxx.x.azuredatabricks.net
resources:
jobs:
my_job_name:
tasks:
- task_key: ingestion
libraries:
- whl: ./dist/*.whl
- pypi:
package: some-package==${var.some_package_version}
repo: https://pkgs.dev.azure.com/stihl/xxxx/_packaging/xxx/pypi/simple/
- task_key: preparation
libraries:
- whl: ./dist/*.whl
- pypi:
package: some-package==${var.some_package_version}
repo: https://pkgs.dev.azure.com/stihl/xxxx/_packaging/xxx/pypi/simple/
I was able to achieve this with yaml anchoring. It feels a bit jenky, but seems to work and cuts down on that boilerplate.
I will leave only the important parts of the yaml in the answer here. The important part is the libs
property, and how its referenced under the task_key
.
---
# deployment.yml
---
resources:
jobs:
example-dab-workflow:
libs: &libs
libraries:
- pypi:
package: llama-index==0.10.43
- pypi:
package: pytest==8.3.2
- pypi:
package: transformers==4.41.1
- whl: ../dist/*.whl
name: "example-dab"
tasks:
- task_key: example_task
existing_cluster_id: abc1234-foo999-barre
python_wheel_task:
package_name: example_dab
entry_point: "example_entrypoint"
<<: *libs