google-cloud-platformgoogle-bigquerygoogle-colaboratoryspring-cloud-gcp-bigquery

How do I save a table from big query into colab as a pandas dataframe?


I am new to BQ I am trying to save my BQ tables as a pandas dataframe in my colab environment. This is the code I am using but I am getting a "Bad request Error". Any ideas how I can trouble shoot? I can't figure out what I am doing wrong. My code is as follows:

from google.cloud import bigquery

client = bigquery.Client(project=project_id)

sample_count = 2000
row_count = client.query('''
  SELECT 
    COUNT(*) as total
  FROM `123.cleaned_sales`''').to_dataframe().total[0]

df = client.query('''
  SELECT
    *
  FROM
    `123.cleaned_sales`
  WHERE RAND() < %d/%d
''' % (sample_count, row_count)).to_dataframe()

print('Full dataset has %d rows' % row_count)```

Here is my error message 

[enter image description here][1]


  [1]: https://i.sstatic.net/VQOox.png

Solution

  • The image you shared indicates the projectId and datasetId should not to be empty in this query :

    row_count = client.query('''
      SELECT 
        COUNT(*) as total
      FROM `123.cleaned_sales`''').to_dataframe().total[0]
    

    You have to set the projectId :

    row_count = client.query('''
      SELECT 
        COUNT(*) as total
      FROM `your_project_id.123.cleaned_sales`''').to_dataframe().total[0]
    

    Do the same with the second query.