pysparkazure-synapsespark-notebookazure-notebooks

Pyspark - %Run magic command in Synapse Notebook: pass dynamic parameter

In synapse notebook, I'm using the %run magic command to running a notebook and it is running well:

%run "5 Bronze to silver/Daily/NB_B2S" { "exclude_bronze_notebook_names": '["NB_B2S_SNOW_Task", "NB_B2S_Time_and_Resources_Task_Time"]' }

but when I tried to to pass the values dynamically trough parameter, for example:

bronze_notebook_names = ['NB_B2S_SNOW_Task', 'NB_B2S_Time_and_Resources_Task_Time']

%run "5 Bronze to silver/Daily/NB_B2S" { "exclude_bronze_notebook_names": bronze_notebook_names }

It returns an error:

MagicUsageError: Cannot parse notebook parameters. More details please visit https://go.microsoft.com/fwlink/?linkid=2173018 --> JsonReaderException: Unexpected character encountered while parsing value: b. Path 'exclude_bronze_notebook_names', line 1, position 35.

The alternative would be to use mssparkutils.notebook.run(), but I want to use %run.

Is there any way to achieve this with %run magic command?

Solution

  • You will have to use the mssparkuntils command. The %run magic command does not directly support passing dynamic parameters to the script or notebook being run.