apache-sparkpysparkapache-spark-sqlapache-iceberg

Providing table options while writing to iceberg table using spark

I am trying to write to a iceberg table(which does not exist before write, hence creating during write) and would like to provide few table properties. Is it possible to do so using the dataframeWriter? I do not want to fire a sql query using spark.sql()

the following are some of the configs that I am using.

"spark.sql.catalog.spark_catalog": "org.apache.iceberg.spark.SparkSessionCatalog"

"spark.sql.extensions": "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions"

"spark.sql.catalogImplementation": "hive"

Solution

  • Using DataFrameWriterV2, this is possible:

    spark.range(10).withColumn("tmp", lit("hi")).writeTo("test.sample").using("iceberg").tableProperty("write.spark.accept-any-schema", "true").createOrReplace()