I am trying to write to the glue catalog from my glue script. However, I would like to overwrite the data and not append. I have this piece of ode
def data_export(df, target_db, target_table):
additional_options = {
"enableUpdateCatalog": True,
"updateBehavior": "UPDATE_IN_DATABASE"
}
try:
dynamic_frame_out = DynamicFrame.fromDF(df, glue_context, "mydb")
glue_context.write_dynamic_frame.from_catalog(
frame=dynamic_frame_out,
database=target_db,
table_name=target_table,
additional_options=additional_options
)
except Exception:
raise ("Error")
I am not able to figure out what additional options would allow me to overwrite the data every time I run my script.
Dynamic Frame unfortunately does not support overwrite mode. So what you are trying is not really possible, you would need to switch to native Spark instead.