I'm currently working on extracting data from a REST API using Talend's tRest and tExtractJSONFields components. However, I'm facing an issue where I'm only able to extract a maximum of 25 rows, despite the fact that the dataset contains many more entries.
Here's the workflow I'm using, and I've noticed that regardless of the queries I try, I'm consistently limited to 25 rows. I suspect there might be a pagination or limit setting that I'm missing.
Could anyone guide me on how to navigate the JSON to effectively extract more than 25 rows, and bypass the pagination ? Any suggestions or insights would be greatly appreciated!
So first you need to extract the number of objects there is the Rest API and use it to get the number of iteration for the tRest.
So the job would look like this :
In the first tExtractJSONFields you retreive the count value like this :
In the tJavaRow you divide that value by 25 and round it up like this :
And store that value in a globalVar, I used the tFlowToIterate to do that.
The you use a tLoop with the configuration like this :
To navigate your paginated data in the API and I store it in a tHashOutput.
Hope this helps