google-cloud-platformgoogle-datastream

Datastream created table and overwrites historic data


I have a datastream between a PostrgreSQL and Google BigQuery which was running smoothly over the last couple weeks. Suddenly all historic data was gone and only incremental data was added again. I've check the activity and found that the Google key created all the tables again (and probably replaced existing ones), but without historic backfill. Does anyone knows when this happens and how to prevent it?

Thanks!

Activity logs


Solution

  • Problem solved, it was because of the expiry date on the tables that is 60 days by default.