google-bigquerystreamingaws-rds-data-service

how to load data from AWS RDS to Google BigQuery in streaming mode?

how to load data from AWS RDS to Google BigQuery in streaming mode? Description: I have data in RDS (SQL Server), and wanted to load this data into Google BigQuery in real-time.

Solution

  • There is no direct way to insert changes from Amazon RDS to Google Cloud BigQuery. It could be done with a pipeline like this

    Amazon RDS ----Lambda/DMS----> Kinesis Data Streams -----Lambda----> BigQuery

    1. Read changes from Amazon RDS to Kinesis Data Streams using Lambda or use Cloud DMS. You can also push it to Kinesis Firehose for aggregating/batching records.
    2. Use Lambda to read from Kinesis streams/Firehose to insert into BigQuery using tabledata.insertAll (BQ streaming API). Code will be something similar to this.