databricksazure-databricksaws-databricks

Azure databricks cluster init script using workspace path: Init script failure: RESOURCE DOES NOT EXIST

I am trying to add a simple init script to my Azure databricks cluster. The script path is like /Workspace/Shared/my-cluster-init.sh. When I tried to start the cluster with the init script, I am seeing the following error:

Init script failure: RESOURCE DOES NOT EXIST: No file found at /Workspace/Shared/cluster-init.sh

Which is confusing, because the file does exist. I have tried with different access mode (Single user, shared, No isolation shared), and they all fail with the same error.

My databricks runtime version: 10.4 LTS (includes Apache Spark 3.2.1, Scala 2.12).

Solution

  • You need to remove /Workspace part, just leave /Shared/my-cluster-init.sh - as per documentation:

    For an init script stored in your home directory with workspace files: /Users/<user-name>/<script-name>.sh.