apache-sparkpysparkapache-spark-sqlamazon-emrapache-iceberg

pyspark & iceberg: `update *` not working in `merge into`?

I run pyspark on AWS EMR studio

MERGE INTO iceberg_catalog.staging.tbl AS t
USING tempview AS s
ON t.number = s.number
WHEN MATCHED THEN UPDATE *
WHEN NOT MATCHED THEN INSERT *

but got

pyspark.sql.utils.ParseException:
Syntax error at or near '*'

Any idea why? Anything missing here?

Solution

  • It seems that the update statement should be used with the update set syntax. This is an example of an Iceberg table used in AWS Athena, but please refer to it.