Sink all projects under an organisation into a bigtable
I am building a similar approach to bring all the logs into BigQuery dataset. As per this https://dev.to/alvardev/gcp-cloud-logging-the-basis-45eh I am able to bring the project log details into the corresponding BigQuery. But how do we bring all the projects logs into a single Bigtable i.e for example lets say An organisation has 50 projects and with minimal configuration I want all the logs from 50 projects to come under a single Bigtable as consolidated project logs.
Is this possible in GCP, can someone please help me on this
Currently, the only supported destinations based on the documentation are Cloud Storage, Pub/Sub, BigQuery and another Cloud Logging Bucket . However I found this another approach where the architecture is: Cloud Logging -> Pub/Sub -> DataFlow -> BigTable.
Basically, you will create a sink and choose Pub/Sub topic as the destination then from there, you can stream your data from Pub/Sub to Bigtable using Dataflow. You can check this SO answer that you can use as reference on how to stream data from Pub/Sub to BT.
- oauth 2 parameters can only have a single value: scope
- Correct use of gcloud --sort-by combined with --limit
- App attestation failed with Firebase App check in release on Android
- Why doesn't the offline cloud firestore documentation have code for offline query indexes for flutter?
- _CHANGE_TYPE not working when streaming Google Big Query rows from Pub/Sub
- Migrating from legacy Google FCM to HTTP v1 for push notifications
- Google OAuth 2 Error 400: redirect_uri_mismatch but redirect uri is compliant and already registered in Google Cloud Console
- Can we update a field due to its value without reading it in firebase?
- Use process.env variables inside next.config.js
- What is the best practice for decoding Firestore documents in a listener for Swift 5?
- How to install Google Cloud SDK app-engine-python on Debian 12?
- Bigquery service account restricted to a dataset
- Gcloud Pub/Sub: Total timeout of API google.pubsub.v1.Publisher exceeded 60000 milliseconds before any response was received
- Keep getting the error Google Sign-In Error: PlatformException(sign_in_failed, com.google.android.gms.common.api.ApiException: 10: , null, null)
- Error 400 invalid JSON Payload Unknown name generationConfig on an AI API Curl command
- How to access cloud run environment variables in Dockerfile
- Error 400: Invalid JSON payload received when calling Google Generative Language API
- GCP Bucket Permissions by Prefix
- database connection URI string for GCP cloudsql postgres database?
- Who control Google Cloud platform filtering?
- gcloud cli command create eventarc errors
- Is there a way to block HTTP on GCP AppEngine
- How to generate GOOGLE_CLIENT_ID and GOOGLE_CLIENT_SECRET without a domain?
- How to change the source GitHub repository for a deployed project on Google Cloud Run?
- Not able to access Nexus console on my Ubuntu VM instance in GCP
- VertexAI Tabular AutoML rejecting rows containing nulls
- How to access Google Cloud Speech-to-text v2 API through HTTP/REST
- Importing a Model with Scikit Learn on Vertex
- Is there a simple way to host a JSON document you can read and update in Google Cloud Platform?
- how can I get my gcloud user creds into a container securely and use them to impersonate a service account when testing locally?