amazon-web-serviceshiveaws-glueaws-glue-data-catalog

Save Data to AWS Glue via Glue Script


I am trying to write to the glue catalog from my glue script. However, I would like to overwrite the data and not append. I have this piece of ode

def data_export(df, target_db, target_table):
additional_options = {
    "enableUpdateCatalog": True,
    "updateBehavior": "UPDATE_IN_DATABASE"
}
try:
    dynamic_frame_out = DynamicFrame.fromDF(df, glue_context, "mydb")
    glue_context.write_dynamic_frame.from_catalog(
        frame=dynamic_frame_out,
        database=target_db,
        table_name=target_table,
        additional_options=additional_options
    )
except Exception:
    raise ("Error")

I am not able to figure out what additional options would allow me to overwrite the data every time I run my script.


Solution

  • Dynamic Frame unfortunately does not support overwrite mode. So what you are trying is not really possible, you would need to switch to native Spark instead.