google-cloud-platformgoogle-bigquerygoogle-cloud-pubsubterraform-provider-gcp

Google cloud - pub/sub - write to Bigquery with metadata - missing fields


I'm trying to enable the Dead letter topic by creating a new topic, called dl_topic, and a subscription to this topic called dl_subs. For the dl_subs subscription, I'm enabling the Write to BigQuery option, so I will have the data saved. I would like also to forward metadata of the topic to BQ. If I select the Write metadata option, Google suggests also adding a few other fields in the BQ.

google_bigquery_table
message_id
publish_time
data
attributes

I'm doing this in Terraform. Question is:

also, after the dead_letter topic receives a message and is pushed to BigQuery, how do I replay the message (from BQ)?


Solution

  • From terraform and GCP documentation it seems you need these additional column in your BigQuery Table.

    Terraform documentation:

    enter image description here

    GCP documentation:

    Write metadata

    This option lets Pub/Sub write the metadata of each message to additional columns in the BigQuery table. Else, the metadata is not written to the BigQuery table.

    If you select the Write metadata option, ensure that the BigQuery table has the fields described in the following table.

    So these additional field will go in BQ schema where you are creating BQ Table and in your terraform code add write_metadata = true.