I've established a workflow in Appflow that relies on a connection to BigQuery. However, when I execute the workflow, I come across an issue outlined below:
The request failed because the service Source Google BigQuery returned the following error: Details: The response size of 14.50 MB is too large. Responses can be up to 5.50 MB., ErrorCode: ClientError.
According to bigquery response limits the response could be in GB's. But in my case it fails with only 14.5MB.
When you use Google BigQuery as either the source or destination, you can configure the Google BigQuery API preference setting. Use this setting to specify whether Amazon AppFlow uses synchronous (smaller data transfers) or asynchronous (larger transfers) data transfer when you run your flow.
The Amazon AppFlow console provides this setting on the Configure flow page under Source details or Destination details. To view it, expand the Additional settings section.
You can choose one of these options:
Automatic (default) — For each flow run, Amazon AppFlow selects the type of data transfer to use.
Standard — Amazon AppFlow uses only Google BigQuery synchronous data transfer. This option optimizes your flow for small to medium-sized data transfers.
Bulk — Amazon AppFlow runs Google BigQuery asynchronous data transfers, and it's optimal for large datasets.
Refer to AWS documentation Link here