google-bigqueryinformation-schema

Are bytes processed in Quotas & System Limits or INFORMATION_SCHEMA.JOBS_BY_PROJECT used for billing?

I'm seeing a significant discrepancy between two different ways of measuring BigQuery usage:

  1. In the Google Cloud Console's "Quotas & System Limits" page, the chart shows approximately 125GB processed over the last 30 days
  2. Querying INFORMATION_SCHEMA.JOBS_BY_PROJECT shows only about 20GB processed for the same period

Infromation_Schema

Quotas and Limits

Questions:

  1. Which of these numbers is used for billing purposes?
  2. Why is there such a large difference between these two measurements?
  3. Which should I rely on when trying to stay within the 1TB free tier limit?

Thanks!!

Sample query used:

SELECT
  DATE_TRUNC(DATE(creation_time), MONTH) as month,
  ROUND(SUM(total_bytes_processed)/POW(2,30), 2) AS gigabytes_processed
FROM 
  `region-asia-east1`.INFORMATION_SCHEMA.JOBS_BY_PROJECT
WHERE 
  creation_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 30 DAY)
GROUP BY 
  month
ORDER BY 
  month DESC;
Solution

  • The Quotas & Limits measures all the data scanned by queries, including cached queries or jobs that don't include billing. It also tracks the logical_bytes_processed for monitoring against usage limits.

    Whereas, the INFORMATION_SCHEMA.JOBS_BY_PROJECT tracks the query level details including logical bytes and the bytes billed after optimization. You need to rely on total_bytes_billed in the Information_schema to get idea of billing which comes under on-demand pricing.

    You can check this documentation for more details regarding On-demand pricing other pricing in BigQuery.

    As per this documentation, BigQuery offers some resources free of charge up to a specific limit. It will be during and after the free trial period. If you go over these usage limits and are no longer in the free trial period, you will be charged accordingly.