[SOLVED] SQL Widgets in Databricks Interactive Serverless Compute Clusters Not Working

SQL Widgets in Databricks Interactive Serverless Compute Clusters Not Working

I have the following code in a Notebook that creates a widget with three options.

create widget dropdown environment default 'int' choices
select * from (values ('int'), ('stage'), ('prod'));

This runs perfectly fine on a personal all purpose compute cluster. However, after switching to serverless compute for notebooks, I get the following error:

DriverException: Selection sequence must include int
File <command-2382736690910233>, line 1
----> 1 get_ipython().run_cell_magic('sql', '', "create widget dropdown environment default 'int' choices select * from (values ('int'), ('stage'), ('prod'));\n")
File /databricks/python_shell/dbruntime/sql_magic/sql_magic.py:163, in SqlMagic.sql(self, line, cell)
    161     break
    162 elif request.get("status") == "error":
--> 163     raise DriverException(request.get("message"))
    164 else:
    165     # this should never happen
    166     raise Exception("Unknown comm message received: " + str(request))

Is this a general problem with serverless notebook clusters?

Python widgets do work though. However, I can't use them with the remainder of my notebook, which is SQL.

dbutils.widgets.dropdown("environment", "int", ["int", "stage", "prod"])

Solution

Widgets can only be used in standard clusters and not in serverless clusters. The documentation on https://docs.databricks.com/en/notebooks/widgets.html#create-widgets-with-sql-python-r-and-scala also says:

Programmatically create widgets in a notebook attached to a compute cluster.

These interactive features (i.e. widgets) are built on top of the Apache Spark environment. However, serverless clusters do not provide a "full" Spark environment.