jsontalend

Extracting more than 25 rows from REST API using tRest and tExtractJSONFields


I'm currently working on extracting data from a REST API using Talend's tRest and tExtractJSONFields components. However, I'm facing an issue where I'm only able to extract a maximum of 25 rows, despite the fact that the dataset contains many more entries.

Here's the workflow I'm using, and I've noticed that regardless of the queries I try, I'm consistently limited to 25 rows. I suspect there might be a pagination or limit setting that I'm missing.

Could anyone guide me on how to navigate the JSON to effectively extract more than 25 rows, and bypass the pagination ? Any suggestions or insights would be greatly appreciated!

Here is the workflow and my count : count of JSON rows


Solution

  • So first you need to extract the number of objects there is the Rest API and use it to get the number of iteration for the tRest.

    So the job would look like this : enter image description here

    In the first tExtractJSONFields you retreive the count value like this :

    enter image description here

    In the tJavaRow you divide that value by 25 and round it up like this :

    enter image description here

    And store that value in a globalVar, I used the tFlowToIterate to do that.

    The you use a tLoop with the configuration like this :

    enter image description here

    To navigate your paginated data in the API and I store it in a tHashOutput.

    Hope this helps