databricksazure-databrickscross-joindatabricks-sqlcross-apply

Is there a way to use CROSS APPLY from SQL to Spark SQL?

I have complex stored procedure that uses multiple views/functions inside and inside there are multiple cross applies and I am not sure if there is a "easy solution" to replicate it in spark.sql

df = spark.sql(f""" select * from table CROSS APPLY ( some business rule ) """)

I checked the documentation: https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-qry-select-join.html but didn`t found what I want

Solution

  • You must first of all enable this option in your notebook

    spark.sql("SET spark.sql.crossJoin.enabled=true")