I have the following code in a Notebook that creates a widget with three options.
create widget dropdown environment default 'int' choices
select * from (values ('int'), ('stage'), ('prod'));
This runs perfectly fine on a personal all purpose compute cluster. However, after switching to serverless compute for notebooks, I get the following error:
DriverException: Selection sequence must include int
File <command-2382736690910233>, line 1
----> 1 get_ipython().run_cell_magic('sql', '', "create widget dropdown environment default 'int' choices select * from (values ('int'), ('stage'), ('prod'));\n")
File /databricks/python_shell/dbruntime/sql_magic/sql_magic.py:163, in SqlMagic.sql(self, line, cell)
161 break
162 elif request.get("status") == "error":
--> 163 raise DriverException(request.get("message"))
164 else:
165 # this should never happen
166 raise Exception("Unknown comm message received: " + str(request))
Is this a general problem with serverless notebook clusters?
Python widgets do work though. However, I can't use them with the remainder of my notebook, which is SQL.
dbutils.widgets.dropdown("environment", "int", ["int", "stage", "prod"])
Widgets can only be used in standard clusters and not in serverless clusters. The documentation on https://docs.databricks.com/en/notebooks/widgets.html#create-widgets-with-sql-python-r-and-scala also says:
Programmatically create widgets in a notebook attached to a compute cluster.
These interactive features (i.e. widgets) are built on top of the Apache Spark environment. However, serverless clusters do not provide a "full" Spark environment.