terraformdatabricksazure-databricksterraform-provider-azureterraform-provider-databricks

Change Azure Databricks cluster's runtime version to 13.3 LTS by terraform


I have created a Databricks cluster in an Azure Databricks workspace by using terraform. I am using terraform azurerm and databricks providers. The runtime version of my cluster is 11.3 LTS. Now I would like to change it to 13.3 LTS. All my changes to the cluster must be done by terraform and not manually. This is our policy at our organization. Here is the terraform code I use to create the cluster: I have made a module and I call the module in my code which worked very well for ruuntime 11.3:

mod_databricks_cluster.tf file is as follows:

module "mod_cluster_job_run" {
  source                    = "git::https://dev.azure.com/MyOrganization/MyProject/_git/tf-module-databricks-cluster?ref=2.0.0"
  cluster_name              = "tf-spark-job-cluster"
  autoscale                 = false
  autotermination_minutes   = 60
  spark_version             = "11.3.x-scala2.12" #"13.0-lts"#"13.0.x-scala2.12"
  spark_conf = local.spark_conf_test
  node_type_id = var.cluster_node_type
  driver_node_type_id = var.cluster_node_type
  num_workers = var.num_workers
}

This works perfectly well. But when I try to change spark_version to any of these values "3.12.0-scala2.12", "3.12.1-scala2.12", "3.12.2-scala2.12", "13.0-lts" or "13.0.x-scala2.12", it fails with the following error: (terraform apply is running in an Azure DevOps pipeline)

Error: cannot update cluster: Invalid spark version 3.12.0-scala2.12. with module.mod_cluster_job_run.databricks_cluster.cluster, on .terraform/modules/mod_cluster_job_run/main.tf line 1, in resource "databricks_cluster" "cluster": 1: resource "databricks_cluster" "cluster"

Here is the screen shot of the error:

Terraform Apply error

I get the exact same error with other values that I wrote above.

My question is what spark version should be in used in terraform to change the runtime to 13.3 LTS? In other words to change from the existing runtime shown below: The AS-IS cluster

To the runtime 13.3 LTS shown below:

Runtime version 13.3 LTS

What spark version value is acceptable for terraform?


Solution

  • You can get all valid Spark version from Databricks Cluster API or CLI:

    $ databricks clusters spark-versions | jq -r '.versions[].key' | sort
    (...)
    13.3.x-cpu-ml-scala2.12
    13.3.x-gpu-ml-scala2.12
    13.3.x-photon-scala2.12
    13.3.x-scala2.12
    (...)
    

    13.3.x-scala2.12 and 13.0.x-scala2.12 are valid.

    3.12.0-scala2.12, 3.12.1-scala2.12, 3.12.2-scala2.12, 13.0-lts are not.