google-bigquerygoogle-bigquery-storage-api

Retention Period for Uncommitted Data in BigQuery Write Streams with Pending Type

I'm using BigQuery's write stream with the pending type, and I'm wondering what happens to data that hasn't been committed if the stream is finalised (Or due to runtime exception).

// This query fail : 
client.batchCommitWriteStreams(commitRequest);

Could someone tell me how long this uncommitted data sticks around before it gets deleted automatically?

Solution

  • As per BigQuery Storage Write API - Batch load data using pending type article suggests:

    After the stream is finalized and before it is committed, the data can remain in the buffer for up to 4 hours. Pending streams must be committed within 24 hours.

    Once a stream is finalized, it can only stay buffered up to 4 hours.