google-cloud-platformgoogle-bigqueryservicenowservicenow-rest-api

ServiceNow data (API) to Google Cloud Platform

First of all, sorry if I don't explain everything super technical, I am doing my best.

I have ServiceNow REST API (a link with credentials that pulls data daily in a JSON format).

I want to use a product in Google Cloud Platform (GCP) that pulls the data weekly or daily, and stores it in a GCP product. After that, I want to wrangle the data, perhaps in another GCP product.

My questions:

  1. Which GCP product should I use to pull ServiceNow data?
  2. How can I do it? Do I need to code it or just use something like Dataflow?
  3. Which GCP product should I use to store & wrangle the data?

Thanks for the help

Solution

  • What you need in order of execution is this:

    1. Setup Secret Manager entry with the API credentials you will use to connect and authorize with ServiceNow API.

    2. You can use Cloud Workflows to obtain data from ServiceNow API. Workflows has a snippet to pull the authorization key from Secret Manager. Also it integrates with any public API even outside of GCP, so you can connect and use ServiceNow API.

    3. You can later extend your Cloud Workflow to have many steps and you can write your data to BigQuery. Why BigQuery? Because it integrates nicely with other services like Dataprep for data wrangling, or any other pure code solution like Dataflow or Cloud Functions.

    4. To invoke your Cloud Workflows regularly you can setup a Cloud Scheduler.