ValueError: Unterminated string starting at: Error while calling an API endpoint recursively in Python 3.7

I am trying to connect to an API endpoint, fetching the json response, writing and extending the response (looping through page: total number of pages are 200, number of items: 41500) to a list. And finally I would like to write the complete list to a single JSON file. I am running below code for performing this task.

url = "https://abcd../v1/org"
headers = {'Accept': 'application/json'}
params = {'pagesize': '200', 'versions' : 'true'}

organisations = []
new_results = True
page = 1

while new_results:
    org_url = url + f"?page={page}"
    response_api = requests.get(org_url, auth=('asdff', '364gbudgsh$'), params = params, headers = headers).json()
    organisations.extend(response_api)
    page += 1
    
with open('organisations.json', 'w') as f:
    for line in organisations:
        f.write(f"{line}\n")

However, after some API calls, I am getting this error.

ValueError: Unterminated string starting at:

Any suggestion to improve the API call or to solve this error would be really helpful.

Solution

I think you should check the API documentation and/or debug what and how are you getting the data. But I hope this can help you:

url = "https://abcd../v1/org"
headers = {'Accept': 'application/json'}
params = {'pagesize': '200', 'versions': 'true'}  # TODO: Check the API, is it possible to get more than one data in a single page?

user = os.getenv('API_USER')  # TODO: Save them somewhere safely, don't hardcode the user and password
psw = os.getenv('API_PSW')

organisations = []
new_results = True
page = 1

while new_results:
    org_url = url + f"?page={page}"
    response_api = None
    try:
        response_api = requests.get(org_url, auth=(user, psw), params=params, headers=headers).json()
        if not response_api['status'] == 200:  # TODO: Check in the API doc or debug how exactly is named the status
            new_results = False
        organisations.extend(response_api['data'])  # TODO: Check the data or the message
    except Exception as e:
        print(f'Debug here. Exception: {e}.\nresponse_api is: {response_api}')
    page += 1

with open('organisations.json', 'w') as f:
    for line in organisations:
        f.write(f"{line}\n")