pythonazure-data-factorydatabricksazure-databricksspark-notebook

databricks Python notebook from azure data factory or locally if statement

I have a Databricks Python notebook that reads in a parameter from ADF using:

Program_Name = dbutils.widgets.get("Program_Name")

Is there an IF statement or something similar I can do in the notebook code, such that when I run the notebook interactively, it will substitute the call to dbutils with a plain assignment? Logically I want something like:

if running in ADF:
   Program_Name = dbutils.widgets.get("Program_Name")
else:
   Program_Name = 'ABC123'

If such a thing is possible, it beats the alternative of having to comment out the dbutils call every time I modify the rest of the notebook :) I've done similar things so that a script can be run from Jupyter/PyCharm or from the command line, but am not aware of anything that tells the python interpreter it's been called from ADF.

Many thanks!

Solution

  • Instead of catching up exception, it's just easier to create widget explicitly, and set default value (see docs).

    dbutils.widgets.text("Program_Name", "ABC123", "Program name")
Program_Name = dbutils.widgets.get("Program_Name")

    This has following benefits:

    1. The code is simpler - you don't need to have do any try/catch
    2. If necessary you can pass another program name even if you run notebook interactively