I am working with API data from the US Treasury. I used the GET() function from the httr package and fromJSON() function from jsonlite package. Here is an example API:
library(httr)
library(jsonlite)
url <- https://api.fiscaldata.treasury.gov/services/api/fiscal_service/v2/accounting/od/avg_interest_rates
data1 <- GET(url)
data2 <- fromJSON(rawToChar(data1$content))
data2 <- data.frame((data1$data))
The problem is I only see data for the first 100 observations. Output:
str(data3)
'data.frame': 100 obs. of 11 variables:
$ record_date : chr "2001-01-31" "2001-01-31" "2001-01-31" "2001-01-31" ...
$ security_type_desc : chr "Marketable" "Marketable" "Marketable" "Marketable" ...
$ security_desc : chr "Treasury Notes" "Treasury Bonds" "Treasury Inflation-Indexed Notes" "Treasury Inflation-Indexed Bonds" ...
$ avg_interest_rate_amt : chr "6.096" "8.450" "3.772" "3.866" ...
$ src_line_nbr : chr "2" "3" "4" "5" ...
$ record_fiscal_year : chr "2001" "2001" "2001" "2001" ...
$ record_fiscal_quarter : chr "2" "2" "2" "2" ...
$ record_calendar_year : chr "2001" "2001" "2001" "2001" ...
$ record_calendar_quarter: chr "1" "1" "1" "1" ...
$ record_calendar_month : chr "01" "01" "01" "01" ...
$ record_calendar_day : chr "31" "31" "31" "31" ...
I need the whole available data (not just the first 100 observations) and I am not able to figure out where the issue is. The API itself does not restrict or impose limits on data. Can you please help me out?
As hinted in my comment above, this is a server-side restriction. The API docs at https://fiscaldata.treasury.gov/datasets/average-interest-rates-treasury-securities/average-interest-rates-on-u-s-treasury-securities describe pagination and e.g. by requesting
url <- paste0("https://api.fiscaldata.treasury.gov/",
"services/api/fiscal_service/v2/accounting/od/avg_interest_rates")
res1 <- RcppSimdJson::fload(paste0(url,"?page[number]=1&page[size]=1000"))
res2 <- RcppSimdJson::fload(paste0(url,"?page[number]=2&page[size]=1000"))
you request and get pages one and two of sets sized at a 1000 records.