I wanted to create a bunch of notebooks on my azure databricks workspace using terraform. All the necessary python files needed to create those notebooks are locally available under the notebooks folder under the name 'Vendor_Consolidation' with the following folder structure:
Advanced_analytics
- CICD
- Terraform
- notebooks.tf
- output.tf
- variables.tf
- notebooks
- vendor_consolidation
- import_GW
- import_gw_notebook.py
- import_bc
- import_bc_notebook.py
- common_code
- print_data_notebook.py
- consolidate_notebook.py
- validate_copy_notebook.py
The same local folder strucutre has to be maintained once i copy the notebooks over to the workspace under a folder called 'live'.
My notebooks.tf code looks like below:
resource "databricks_notebook" "print_vendor"{
for_each = fileset("${path.module}/../../notebooks/vendor_consolidation", "**/*.py")
path = "/Workspace/live/vendor_consolidation/${each.value}"
language = "PYTHON"
}
But my terraform.apply command ran in azure devops pipeline returns the below error (same for all the python files):
╷
│ Error: cannot create notebook: open : The system cannot find the file specified.
│
│ with databricks_notebook.print_vendor["import_GW/import_gw_notebook.py"],
│ on notebooks.tf line 3, in resource "databricks_notebook" "print_vendor":
│ 3: resource "databricks_notebook" "print_vendor"{
│
╵
Is there any issue with my for_each or fileset()? Or am I missing anything in the resource block?
Silly me! I found out the issue with my code. In the notebooks.tf file, i was missing the source argument because of which the file cannot be found error was being thrown. Updated the code as below and the ADO pipeline ran successfully and multiple notebooks got copied created on the databricks workspace.
resource "databricks_notebook" "print_vendor"{
for_each = fileset("${path.module}/../../notebooks/vendor_consolidation", "**/*.py")
path = "/Workspace/live/PrintVendor/${each.key}"
language = "PYTHON"
source = "${path.module}/../../notebooks/vendor_consolidation/${each.key}"
}