paginationazure-data-factory

Pagination rule not working in datafactory


I am trying to get paginated results in datafactory from a rest API. I get the results from the first page correctly and i dont get errors when i run it, i just dont get anything from other pages. I get the response in json like this:

enter image description here

and in datafactory in have it like this: enter image description here

can anyone help me solve this mystery


Solution

  • Your value for pagination is incorrect for your REST API. With your value $.@odata.nextLink, the pipeline run first checks for the key @odata key in your first page response and then if it finds @odata, it will check for nextLink key inside @odata object. If it didn't find the given key @odata, it will stop the pagination and will only give the results till that page i.e., first page which was the same happening in your case as well.

    As the key in your page response containing special characters like dot (.), modify your value like below.

    $['@odata.nextLink']
    

    enter image description here

    This will check for the key "@odata.nextLink" in your page response and stores all data till the last page in your target file.