javarestdremio

Adding a data source via REST call in Dremio

I am trying to create a new data source in Dremio using REST API.

While the docs are pretty decent, they lack real examples which is where I fail.

So wanting to add a new data source I do the following REST calls:

  1. Login to get the session token
  2. Create a datasource: So I can create a folder structure or create a virtual dataset which is based on SQL query of pre-existing data. However I am failing to add a new PHYSICAL_DATASET or promote VIRTUAL_DATASET to a physical one.

I am getting HTTP 400 with a message that the entity to promote with a given path does not exist (when I can verify it does)

Anybody with any experience using Dremio's REST should be able to shed some light.

Thank you!

Solution

  • Can you describe your PHYSICAL_DATASET

    can you add an example of your code and file here

    here example for csv

    1. authenticate to get token

    2. Call POST localhost:9047/api/v3/catalog/{id}

      { "entityType": "dataset", "path": ["REST", "data.csv"], "type": "PHYSICAL_DATASET", "format": { "type": "Text", "fieldDelimiter": ",", "lineDelimiter": "\n", "quote": """, "escape": """, "skipFirstLine": false, "extractHeader": true, "trimHeader": false, "autoGenerateColumnNames": true }

      }

    to get {id} localhost:9047/api/v3/catalog/by-path/SPACE/FILE =>"id": ? {id} should be URLencoded