google-bigquery

Can you create a temporary table as the destination table in BigQuery?


In BigQuery, you can use the bq command line tool to run queries from the command line. However, large queries must have a destination table, and can't directly dump results.

Is there a way to specify a temporary table to be created as the destination table? Attempting to use a non-prefixed table name, which is how temporary tables are normally specified, doesn't work:

bq query  --nouse_legacy_sql --session_id=$sid --format=prettyjson --destination_table tempout  'select 1'
BigQuery error in query operation: Invalid value tempout for destination_table: Cannot determine table described by tempout

The official docs have an example using a regular table but not one with a temporary table. It seems temporary tables (like in sessions) cannot be used this way, but it isn't explicit.


Solution

  • You can use a dataset with a default table expiration, so all tables will act as temporary tables:

    bq mk --location=US --dataset --default_table_expiration=3600 project:temp_dataset
    

    Another option is update the expiration of the created table:

    bq query  --nouse_legacy_sql --session_id=$sid --format=prettyjson --destination_table project:dataset.tempout 'select 1'
    bq update --expiration 3600 project:dataset.tempout