google-cloud-platformairflowdirected-acyclic-graphs

How to Transfer a BigQuery view to a Google Cloud Storage bucket as a csv file


I need to export the content of a BigQuery view to the csv file in GCP, with Airflow DAG. To export the content of the BQ TABLE, I can use BigQueryToCloudStorageOperator. But in my case I need to use an existing view, and BigQueryToCloudStorageOperator fails with this error, which I see while checking the logs for failed DAG:

BigQuery job failed: my_view is not allowed for this operation because it is currently a VIEW

So, what options do I have here? I can't use a regular table, so may be there is another operator that would work with a view data stored in BQ, instead of table? Or may be the same operator would work with some addition options (although I don't see anything useful in here Apache documentation for BigQueryToCloudStorageOperator)?


Solution

  • I think the Bigquery client doesn’t give the possibility to export a view to a GCS file.

    It’s not perfect but I propose you 2 solutions

    First solution (more native with existing operators) :

    Second solution (less native with Python clients and PythonOperator) :

    I have a preference for the first solution, even if it forces me to create a staging table.