How to filter and extract data from Qlik Sense JSON-RPC API

I have successfully been able to connect to the Qlik Sense JSON-RPC API (QIX) using websockets. And I'm able to see all of the tables, fields, and table data. The tables have millions of rows and querying through all of the data takes hours. I want to be able to filter by one of the timestamp fields to only get the most recent data.

Here are the API calls I'm currently making after connecting to wss://{tenant}.us.qlikcloud.com/app/{app_id}

{
  "jsonrpc": "2.0",
  "id": 1,
  "handle": -1,
  "method": "OpenDoc",
  "params": {
    "qDocName": "{app_id}"
  }
}

{
  "jsonrpc": "2.0",
  "id": 1,
  "handle": 1,
  "method": "GetTablesAndKeys",
  "params": {
    "qWindowSize": {"qcx": 1, "qcy": 1},
    "qNullSize": {"qcx": 1, "qcy": 1},
    "qCellHeight": 1,
    "qSyntheticMode": true,
    "qIncludeSysVars": false,
    "qIncludeProfiling": false
  }
}

{
  "jsonrpc": "2.0",
  "id": 1,
  "handle": 1,
  "method": "GetTableData",
  "params": {
    "qOffset": 0,
    "qRows": 1000,
    "qSyntheticMode": true,
    "qTableName": "{table_name}"
  }
}

How do I GetTableData with a filter applied?

Solution

The general idea is:

make selection in the required field (select the value(s) you need)
create hypercube (session object) with required fields (you can create table in the UI if you want/can and use it as a data placeholder)
"ask" Qlik to send you the data.
once selection is made Qlik will filter all objects in the app (not only the session object) and reduce the data only relevan to your selection. Selections are central concept in Qlik

Example script (JavaScript with Enigma.js)

const valuesToSelect = [
  {
    qText: "Value 1",
  },
  {
    qText: "Value 2",
  },
];

const global = await session.open();
const app = await global.openDoc("some-app-id");
const field = await app.getField("MyField"); // get the field instance
const selectionResult = await field.selectValues(valuesToSelect); // make selections in the field

// prepare the hypercube properties
// specifying the dimensions we want to be displayed
const hypercubeProps = {
  qInfo: {
    qType: "my-hypercube",
  },
  qHyperCubeDef: {
    qDimensions: [
      {
        qDef: { qFieldDefs: ["MyOtherField"] },
      },
      {
        qDef: { qFieldDefs: ["MyYetAnotherField"] },
      },
    ],
    qMeasures: [
      // Measures can be added to the cube as well. If aggregation is needed
      //   {
      //     qDef: { qDef: "=Sum(Value)" },
      //   },
    ],
    qInitialDataFetch: [
      {
        qHeight: 5000,
        qWidth: 2,
      },
    ],
  },
};

const sessionObject = await app.createSessionObject(hypercubeProps); // create our session object
let sessionObjectLayout = await sessionObject.getLayout(); // get its layout. this will populate the data in it

console.log(sessionObjectLayout.qHyperCube.qDataPages); // print the data pages

Hypercubes are used very often in Qlik and are quite important. Especially when there is need to create objects on the fly. Hypercubes have a lot of properties and can be quite messy to deal with them (IMO).

Data pages

{
    qHeight: 5000,    
    qWidth: 2
}

The above specifies how many cells of data to be retrieved initially. These numbers are not random. Qlik allows up to 10 000 cells of data to be returned per request. In our example we asked Qlik to create cube with 2 dimensions. So 2 dimensions * 5000 = 10000 data cells. This also means that if we have more than 10000 data cells we'll have to ask Qlik for the rest of the data. In our case we can use GetHyperCubeData method to retrieve the next page/batch of data (again max 10000 cells/data points)

Paging the data can be tricky especially when we have to deal with pivoted data.

ListBox object

Just to mention that when dealing with single field then might be better/easier to interact with ListBox object instead of cubes/tables. Some JS examples can be found here