cloudetlgoogle-cloud-data-fusiondata-pipeline

İs it possible to create data pipeline with Google Cloud Data fusion using multiple database tables Update or Upsert?

After using multiple database tables plugin and load data to bigquery I would like to make incremental load for every table in one data pipeline.

I wonder if I can use UPSERT on multiple database tables plugin. How can I overcome any advice ?

Solution

  • The UPSERT operation is not currently supported in the BigQuery Multi Table Sink plugin.

    One suggestion is to store records to upsert into another BigQuery Dataset and use a BigQuery Execute Action to execute one merge operation into the desired destination table per each of the source tables.

    See Google BQ - how to upsert existing data in tables? on how to execute the UPSERT operation using SQL.