google-cloud-platformgoogle-data-catalog

How is includeProjectIds used in Google data catalog search?


When searching using Google's data catalog, a scope parameter is required and one of its fields is includeProjectIds. According to the documentation, this parameter represents:

The list of project IDs to search within.

However, the results returned are not limited to that projectid. Am I understanding the parameter incorrectly? I can limit the results using projectid=myproject in the query, but I am trying to understand the includeProjectIds field.

Example request body:

{
  "scope": {
    "includeProjectIds": [
      "MY-PROJECT"
    ]
  },
  "query": "type=dataset"
}

Example response:

{
  "results": [
    # I expect this result:
    {
      "searchResultType": "ENTRY",
      "searchResultSubtype": "entry.dataset",
      "relativeResourceName": "projects/MY-PROJECT/locations/us/entryGroups/@bigquery/entries/....",
      "linkedResource": "//bigquery.googleapis.com/projects/MY-PROJECT/datasets/dataset_name",
      "modifyTime": "2000-01-01T00:00:00Z",
      "integratedSystem": "BIGQUERY",
      "description": "My description"
    },
    # But I don't expect this:
    {
      "searchResultType": "ENTRY",
      "searchResultSubtype": "entry.dataset",
      "relativeResourceName": "projects/NOT-MY-PROJECT/locations/us/entryGroups/@bigquery/entries/....",
      "linkedResource": "//bigquery.googleapis.com/projects/NOT-MY-PROJECT/datasets/dataset_name",
      "modifyTime": "2000-01-01T00:00:00Z",
      "integratedSystem": "BIGQUERY",
      "description": "My description"
    },
    ...
  ]
}

Solution

  • You may try using curl method instead because it uses the service account key of your service account. For service account and keys creation you can refer to this documentation. I tried this and the output was correct having only my project datasets displayed in the results.

    Command:

    -H "Authorization: Bearer "$(gcloud auth application-default print-access-token) \
    -H "Content-Type: application/json; charset=utf-8" \
    -d @request.json \
    "https://datacatalog.googleapis.com/v1/catalog:search"
    

    Inside request.json:

    {
      "scope": {
        "includeProjectIds": [
          "my-project"
        ]
      },
      "query": "type=dataset"
    }
    

    Output:

    a@cloudshell:~ (a)$ curl -X POST \
    -H "Authorization: Bearer "$(gcloud auth application-default print-access-token) \
    -H "Content-Type: application/json; charset=utf-8" \
    -d @request.json \
    "https://datacatalog.googleapis.com/v1/catalog:search"
    {
      "results": [
        {
          "searchResultType": "ENTRY",
          "searchResultSubtype": "entry.dataset",
          "relativeResourceName": "projects/my-project/locations/us/entryGroups/@bigquery/entries/cHJvamVjdHMvdGlwaC1hbmplbGFiL2RhdGFzZXRzLzIwMjIwMTEx",
          "linkedResource": "//bigquery.googleapis.com/projects/my-project/datasets/2",
          "modifyTime": "2022-01-11T01:15:11Z",
          "integratedSystem": "BIGQUERY"
        },
        {
          "searchResultType": "ENTRY",
          "searchResultSubtype": "entry.dataset",
          "relativeResourceName": "projects/my-project/locations/us/entryGroups/@bigquery/entries/cHJvamVjdHMvdGlwaC1hbmplbGFiL2RhdGFzZXRzL0dDUFF1aWNrU3RhcnQ",
          "linkedResource": "//bigquery.googleapis.com/projects/my-project/datasets/G",
          "modifyTime": "2022-01-07T07:54:58Z",
          "integratedSystem": "BIGQUERY"
        },
        {
          "searchResultType": "ENTRY",
          "searchResultSubtype": "entry.dataset",
          "relativeResourceName": "projects/my-project/locations/us-east1/entryGroups/@bigquery/entries/cHJvamVjdHMvdGlwaC1hbmplbGFiL2RhdGFzZXRzLzIwMjExMjI4",
          "linkedResource": "//bigquery.googleapis.com/projects/my-project/datasets/20",
          "modifyTime": "2021-12-28T02:00:47Z",
          "integratedSystem": "BIGQUERY"
        },
        {
          "searchResultType": "ENTRY",
          "searchResultSubtype": "entry.dataset",
          "relativeResourceName": "projects/my-project/locations/us-central1/entryGroups/@bigquery/entries/cHJvamVjdHMvdGlwaC1hbmplbGFiL2RhdGFzZXRzL2JxbWxfdHV0b3JpYWw",
          "linkedResource": "//bigquery.googleapis.com/projects/my-project/datasets/b",
          "modifyTime": "2021-12-14T14:58:09Z",
          "integratedSystem": "BIGQUERY"
        },
        {
          "searchResultType": "ENTRY",
          "searchResultSubtype": "entry.dataset",
          "relativeResourceName": "projects/my-project/locations/us-central1/entryGroups/@bigquery/entries/cHJvamVjdHMvdGlwaC1hbmplbGFiL2RhdGFzZXRzL2JhYnluYW1lcw",
          "linkedResource": "//bigquery.googleapis.com/projects/my-project/datasets/ba",
          "modifyTime": "2021-12-14T14:01:46Z",
          "integratedSystem": "BIGQUERY"
        }
      ]
    }