pysparksnowflake-cloud-data-platform

create snowflake table from spark dataframe


I have my data in s3 location being read into a spark dataframe which I need to push into snowflake. Can snowflake create the table at runtime if the table doesn't exists by the following code?

self.df_source.
write \
.format(SNOWFLAKE_SPARK_CONNECTOR_DRIVER) \
.option("dbtable", f"{self.snowflake_database}.{self.snowflake_schema}. {self.snowflake_table}") \
.options(**self.render_connection_metadata()) \
.option("column_mapping", "name") \
.mode(self.mode) \
.save()

If not how can we create the snowflake table before ingesting the data as we don't know the datatype of each column.


Solution

  • The table gets created if it does not exist