databricksdatabricks-unity-catalog

Write Dataframe to Unity Catalog External Table

I have a unity catalog table called test.dummy. I query and modify the data as follows

df=spark.sql("select * from `test`.dummy")
df=df.where(col("lastname")=="smith")

Now, I would like to use df as my new table. The only way I have found is the following (which requires write file permission):

df.write.mode("overwrite").parquet("abfss://yyy@xxx.dfs.core.windows.net/dd")

Can I update directly the table with the content of df without re-creating the table and without using abffs? I want to use pyspark and just replace the content.

Solution

  • If you use delta, you can do it with the following query:

    df.write.option("overwriteSchema", "true").saveAsTable("`xxx`.`test`.`trips_external`",mode="overwrite")

    With CSV, it does not work. overwriteSchema is only necessary, when the schema changes.