pysparkdatabricksazure-databricksdelta-live-tables

Delta Live Tables clusters termination

I read in the documentation of dlt that "Because Delta Live Tables manages cluster auto-termination and reuse logic, the cluster auto-termination time cannot be overridden."

Is there a way to terminate my dlt cluster when the job is done, from a notebook or the configurations?

Solution

  • You can use Delay compute shutdown configuration for terminating dlt cluster after pipeline execution.

    {
"configuration": {
  "pipelines.clusterShutdown.delay": "60s"
}
}

    Below is without above configuration, cluster is still running after pipeline run also.

    enter image description here

    Set the configuration in dlt settings.

    enter image description here

    Then start the pipeline and click on WAITING_FOR_RESOURCES and select Json.

    enter image description here

    You will get cluster details use cluster id to see the state of it.

    enter image description here

    Output:

    The cluster is terminated after delay of 2 minutes as i given that in configuration.

    enter image description here