I have created a Databricks cluster in an Azure Databricks workspace by using terraform. I am using terraform azurerm and databricks providers. The runtime version of my cluster is 11.3 LTS. Now I would like to change it to 13.3 LTS. All my changes to the cluster must be done by terraform and not manually. This is our policy at our organization. Here is the terraform code I use to create the cluster: I have made a module and I call the module in my code which worked very well for ruuntime 11.3:
mod_databricks_cluster.tf file is as follows:
module "mod_cluster_job_run" {
source = "git::https://dev.azure.com/MyOrganization/MyProject/_git/tf-module-databricks-cluster?ref=2.0.0"
cluster_name = "tf-spark-job-cluster"
autoscale = false
autotermination_minutes = 60
spark_version = "11.3.x-scala2.12" #"13.0-lts"#"13.0.x-scala2.12"
spark_conf = local.spark_conf_test
node_type_id = var.cluster_node_type
driver_node_type_id = var.cluster_node_type
num_workers = var.num_workers
}
This works perfectly well. But when I try to change spark_version to any of these values "3.12.0-scala2.12", "3.12.1-scala2.12", "3.12.2-scala2.12", "13.0-lts" or "13.0.x-scala2.12", it fails with the following error: (terraform apply is running in an Azure DevOps pipeline)
Error: cannot update cluster: Invalid spark version 3.12.0-scala2.12. with module.mod_cluster_job_run.databricks_cluster.cluster, on .terraform/modules/mod_cluster_job_run/main.tf line 1, in resource "databricks_cluster" "cluster": 1: resource "databricks_cluster" "cluster"
Here is the screen shot of the error:
I get the exact same error with other values that I wrote above.
My question is what spark version should be in used in terraform to change the runtime to 13.3 LTS? In other words to change from the existing runtime shown below:
To the runtime 13.3 LTS shown below:
What spark version value is acceptable for terraform?
You can get all valid Spark version from Databricks Cluster API or CLI:
$ databricks clusters spark-versions | jq -r '.versions[].key' | sort
(...)
13.3.x-cpu-ml-scala2.12
13.3.x-gpu-ml-scala2.12
13.3.x-photon-scala2.12
13.3.x-scala2.12
(...)
13.3.x-scala2.12
and 13.0.x-scala2.12
are valid.
3.12.0-scala2.12
, 3.12.1-scala2.12
, 3.12.2-scala2.12
, 13.0-lts
are not.