Python API call pagination issue

I am stuck on the following issue: I have the code below for an API call. In and of itself, the API call is fine, no server issues, I have checked it in Postman:

{
    "@odata.context": "https://api.channeladvisor.com/v1/$metadata#Orders",
    "value": [
        {....}
    ],
    "@odata.nextLink": "https://api.channeladvisor.com/v1/Orders?refresh_token=xxx&$skip=100"
}

I am new to all of this, so please bear with me. All I want to do is save all of the records into a data frame at the end, not just the first 100 records. Channel Advisor's documentation is not really helpful, either, and I haven't been able to find what I need when searching on Google.

I found a post here that I massaged to add on the "while" loop, and I had to modify it to fit my requirements.

Code version 1 (full):

import requests
import pandas as pd
import os
import json

def get_access_token():
    url = 'https://api.channeladvisor.com/oauth2/token'
    headers = {'Authorization': 'Basic xxxxx'}
    post_body = {'grant_type': 'refresh_token', 'refresh_token': 'xxxxx'}
    response = requests.post(url, headers=headers, data=post_body)
    return response.json()['access_token']

access_token = get_access_token()

url = 'https://api.channeladvisor.com/v1/orders'
headers = {
    'Authorization': 'bearer ' + access_token,
    'accept': 'text/plain',
    'Content-Type': 'application/json'
}

response = requests.get(url, headers=headers)
response_data = response.json()['value']
response_nextlink = response.json()['@odata.nextLink']

#create list to store the values
data = []
data.extend(response_data)

while True:
    if '@odata.nextLink' in response_nextlink:
        response = requests.request('GET', response_data['@odata.nextLink'], headers=headers)
        response_data = response.json()['value']
        data.extend(response_data['value'])
        response_nextlink = response.json()['@odata.nextLink']
    else:
        break

This code only runs once, likely because the while loop is not set up correctly.

So, I added some error handling:

Code version 2:

while True:
    response = requests.get(url, headers=headers)
    if response.status_code != 200:
        print("Error occurred. Status code: ", response.status_code)
        print("Content: ", response.content)
        break
    response_data = response.json().get('value')
    if not response_data:
        break
    data.extend(response_data)
    if '@odata.nextLink' in response.json():
        url = response.json()['@odata.nextLink']
    else:
        break

The result of this is a Status code 400 runtime error. Again, the server is fine when I try the call on Postman.

So, I modified the while loop again:

Code version 3:

while True:
    response = requests.get(url, headers=headers)
    response_data = response.json()['value']
    data.extend(response_data)
    if '@odata.nextLink' in response.json():
        url = response.json()['@odata.nextLink']
    else:
        break

This produces the following error:

---------------------------------------------------------------------------
JSONDecodeError                           Traceback (most recent call last)
Input In [80], in <cell line: 25>()
     24 while True:
     25     response = requests.get(url, headers=headers)
---> 26     response_data = response.json()['value']
     27     data.extend(response_data)
     28     if '@odata.nextLink' in response.json():

...................

JSONDecodeError: Expecting value: line 1 column 1 (char 0)

I am otherwise stuck. What am I doing wrong?

Since I dont have API access I can only guess that you confused the break condition in the while loop

import requests
import pandas as pd
import os
import json

# start session to store cookies / authentications
s = requests.Session()

url = 'https://api.channeladvisor.com/oauth2/token'
headers = {'Authorization': 'Basic xxxxx'}
post_body = {'grant_type': 'refresh_token', 'refresh_token': 'xxxxx'}
response = s.post(url, headers=headers, data=post_body)
access_token = response.json()['access_token']

url = 'https://api.channeladvisor.com/v1/orders'
headers = {
    'Authorization': 'bearer ' + access_token,
    'accept': 'text/plain',
    'Content-Type': 'application/json'
}

# use session for requests
response = s.get(url, headers=headers)

# parse response only once
json_response = response.json()

response_data = json_response['value']


#create list to store the values
data = []
data.extend(response_data)

while True:
    if '@odata.nextLink' in json_response:
        response = s.request('GET', json_response['@odata.nextLink'], headers=headers)
        print("request returned status", response.status_code, response.text)
        if not response.ok:
            break
        json_response = response.json()
        data.extend(json_response["value"])
    else:
        break