google-cloud-platformgoogle-bigquerygoogle-cloud-dataprep

Dataprep - Append data to BigQuery table


I'm using GCP's dataprep to join several csv files with the same column structure, treat some data and then write to a BigQuery database.

I have to record this data in BigQuery. Can I include this data from the dataprep and append them in a BigQuery table?


Solution

  • Yes, you can include your data from DataPrep and append them in a BigQuery table.

    Before running the job, in the "Run Job on DataFlow" section:

    1. Click on the action, since you are using BigQuery for the output, should look like "Create-BigQuery"
    2. In the next windows choose you output table
    3. In the left panel select "Append to this table every run"
    4. Click on update

    Now, when you run the Job, this will append your data.

    The following documentation can be useful.