Databricks is smart and all, but how do you identify the path of your current notebook? The guide on the website does not help.

It suggests:

%scala
dbutils.notebook.getContext.notebookPath
res1: Option[String] = Some(/Users/user@org.dk/my_test_notebook)

This does not give me the complete path, but rather the path to some folder structure that is not accessible from the notebook. I need the path, such that I can make system calls in the same folder as the .ipynb file.

Any suggestions?

Access file using Databricks API

I ended up somewhat resolving the problem using the Databricks API to download and upload notebooks and other files to/from Databricks.

1. Read documentation for Databricks Workspace API

Databricks API Documentation

2. Generate API token and Get Notebook path

In the user interface do the following to generate an API Token and copy notebook path:

1. Choose 'User Settings'
2. Choose 'Generate New Token'
3. In Databrick file explorer, "right click" and choose "Copy File Path"

3. Download a Notebook from Databricks

If you want to access a notebook file, you can download it using a curl-call. If you are located inside a Databricks notebook, you can simply make this call either using cell magic, %sh, or using a system call, os.system('insert command').

curl --header "Content-Type: application/json" --request GET --data '{"path":"{/Users/myuser@myorg.com/notebook_to_download}","format":"JUPYTER"}' https://{replace_with_your_databaricks}/api/2.0/workspace/export -H "Authorization: Bearer {my_token}" | jq -r .content | base64 --decode > my_downloaded_notebook.ipynb

4. Uploading a Notebook to Databricks

You can similarly upload a notebook from a machine using the following curl call:

curl -n -F format=JUPYTER -F path="{/Users/myuser@myorg.com/uploaded_notebook}" -F language=PYTHON -F content=@{/my/local/notebook.ipynb} https://{replace_with_your_databaricks}/api/2.0/workspace/import -H "Authorization: Bearer {my_token}"