apache-sparkpysparkapache-spark-sqlapache-iceberg

AnalysisException: mismatched input ';' expecting <EOF>

Trying to load data into Iceberg with pysaprk (version 3.5.1)

%%sparksql

ALTER TABLE
    scheme.table
SET IDENTIFIER FIELDS a, b, c;

ALTER TABLE
    scheme.table
WRITE DISTRIBUTED BY PARTITION LOCALLY ORDERED BY d, e ASC NULLS LAST;

And get this error

AnalysisException:
mismatched input ';' expecting {<EOF>, ',', '.'}
== SQL ==

I tried different combinations of ';', nothing worked.

Solution

  • I found the answer - SparkSQL doesn’t support executing multiple queries with a single call to spark.sql