jsonweb-scrapingxmlhttprequesttruncate

JSON Response is being truncated

I'm trying to scrape some data from a website the data is displayed in a chart ( the data is currency prices over years)

I was able to get the XHR Request and the API link for the JSON data but when I open the response in the ( network tab or in a new tab ) the data is not completely displayed but in the chart the data is represented.

the api link

  1. searched about the problem and I found this post which says that the dev-tools truncates long network response I tried the solution but the same problem is still happening .

  2. tried to use wget to download them but it didn't help same issue appeared.

I'm opening the link in a separate tab on Brave browser (also tried Firefox) I don't know what's the problem

Solution

  • You can scrape that API endpoint (with Python) like below:

    import requests
import pandas as pd

pd.set_option('display.max_columns', None, 'display.max_colwidth', None)

headers = {
    'content-type': 'application/json',
    'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36'
}

r = requests.get('https://sy-exchange-rates-iwi3arxhhq-uc.a.run.app/api/rates?from=Thu,%2031%20Mar%202011%2021:00:00%20GMT&to=Mon,%2023%20Jan%202023%2002:03:23%20GMT&name=USD&source=liranews.info,sp-today.com,dei-sy.com&city=damascus', headers=headers)

df = pd.json_normalize(r.json())
print(df)

    Result in terminal:

        timestamp   source  city    name    buy sell
0   2021-01-03T07:00:00Z    sp-today.com    damascus    USD 2855    2880
1   2021-01-03T07:00:00Z    liranews.info   damascus    USD 2855    2880
2   2021-01-03T07:00:00Z    dei-sy.com  damascus    USD 2845    2855
3   2021-01-03T08:00:00Z    sp-today.com    damascus    USD 2855    2880
4   2021-01-03T08:00:00Z    liranews.info   damascus    USD 2855    2880
... ... ... ... ... ... ...
50772   2023-01-22T22:00:00Z    liranews.info   damascus    USD 6625    6685
50773   2023-01-22T23:00:00Z    sp-today.com    damascus    USD 6625    6685
50774   2023-01-22T23:00:00Z    liranews.info   damascus    USD 6625    6685
50775   2023-01-23T01:00:00Z    sp-today.com    damascus    USD 6625    6685
50776   2023-01-23T01:00:00Z    liranews.info   damascus    USD 6625    6685
50777 rows × 6 columns