I have a set of CRITICAL tables in BigQuery that are getting loaded hourly by DAGS.
I have been tasked to develop a standalone solution to check the following:
Are the tables present ?[ There are chances that the tables may get deleted by operations team]
if The table is present, is the table getting loaded on time ?
If the table is getting loaded, is there a difference in the size during consecutive runs[ The table is expected to increase in size]
If any of the above checks fails, the operations team has to be notified as soon as possible.
Can someone suggest a solution( probably a service or list of services) for the above requirement ?
Part one: Building queries:
Select *
from `region-us`.INFORMATION_SCHEMA.TABLE_STORAGE_USAGE_TIMELINE
For the alerting use one the following:
Select if( condition, ERROR("missing table"),"ok")
Then build a email forwarding in your mail program.