azuregoogle-bigquerydata-ingestion

What is the most efficient way to ingest data from Azure to Bigquery?


I need to do a one-time load (batch) from Azure to BigQuery and I am new in the Google Cloud environment. I noticed there are numerous ways to do this, but still isn't clear which option is the most efficient one.

Any thoughts on this? Thank you

(EDIT)

I didn't have much information on the process when I first came here with the question. Unfortunately, I got a -1 due to the lack of information on the question. You can also comment and ask me to add more information to the question! I now have more info and I'll add a comment with more information in the reply space.


Solution

  • If you are new to GCP, there are two good options for a batch load:

    1. Google Data Transfer, or Storage Data Transfer You can either send the information to the Cloud Storage or BigQuery, there are several options for the source. Documentation: https://cloud.google.com/storage-transfer-service
    2. Google Data Fusion You'll have to set up an instance and then you will be able to create your pipeline. If you are familiar with some ETL tools you will feel at ease to set up the connectors. You can extract the data from various sources and, as the problem states, you can have BigQuery as destination as well. Highly recommend it as a straight-forward solution. Documentation: https://cloud.google.com/data-fusion