sqlpostgresqlgoogle-bigquerygoogle-cloud-sql

How to export data from Google BigQuery to PostgreSQL

I have a table stored in BigQuery and I would like to create a copy of this table+data within a PostgreSQL database. This PostgreSQL is in Google Cloud SQL.

This export will occur daily, that is, the same table will be exported daily to PostgreSQL, so the ideal solution can be automated/repeated, rather than a manual solution.

How can I achieve this?

Solution

  • You can use the BigQuery and CloudSQL APIs to do that, exporting the BQ table as CSV to cloud storage and then importing the file as table in CloudSQL. Then you can automate this process using Cloud Functions + Cloud Scheduler (or even Cloud DataFlow for more complex pipelines).