I'm using Powershell to fetch a list of locations from a vendor site, using an API call:
# Set the initial page and limit values
$page = 1
$limit = 1000
# Initialize an array to hold all the data
$allData = @()
# Set the headers with the token key
$headers = @{
'Content-Type' = 'application/json'
'Authorization' = 'Bearer SomeTokenKey'
'Accept'= 'application/json'
}
# Loop until all data is retrieved
while ($true) {
# Set the API endpoint URL with the current page and limit values
$uri = "https://mysite.Someapisite.com/api/locations?page=$page&limit=$limit"
# Make the API call and retrieve the data
$responseData = Invoke-RestMethod -Uri $uri -Method GET -Headers $headers -Body $body
# Add the retrieved data to the allData array
$allData += $responseData.data
# Check if all data has been retrieved
if ($responseData.meta.pagination.current_page -eq $responseData.meta.pagination.total_pages) {
break
}
# Set the page and limit values for the next API call
$page = $responseData.meta.pagination.current_page + 1
$limit = $responseData.meta.pagination.per_page
}
# Export all the data to a CSV file
$allData | Export-Csv -Path 'c:\extracts\locations.csv' -NoTypeInformation -Force
The vendor has a limitation of extracting 1000 records max per call. The vendor API documentation states that pagination is supported:
The site utilizes an offset/limit based pagination via the limit and page parameters where the offset is automatically calculated based on the parameters provided.
Link: <https://mySite.someAPIsite.com/api/location?page=3&limit=10>; rel="next",
<https://mySite.someAPIsite.com/api/location?page=7&limit=10>; rel="last"
My goal is to extract all records at source, but the above script only returns 1000 records.
(A) Have I not implemented this correctly?
(B) Will using pagination bypass the record limit?
Any thoughts?
I played around the code, and got it do to what I needed:
# API endpoint
$apiUrl = 'https://someSite.siteUrl.com/api/locations'
# API credentials
$accessToken = 'InsertTokenHere'
# Set the limit and initial page
$limit = 1000
$page = 1
# Initialize the locations array
$locations = @()
do {
# Construct the API request URL with pagination parameters
$url = '{0}?limit={1}&page={2}&expand=locationHours' -f $apiUrl, $limit, $page
try {
# Invoke the REST API to fetch locations
$response = Invoke-RestMethod -Uri $url -Headers @{ Authorization = "Bearer $accessToken" }
# Add the fetched locations to the locations array
$locations += $response.data
# Increment the page number for the next API request
$page++
}
catch {
# Output the error message and break out of the loop
Write-Host "API request failed with error: $_"
break
}
} until ($response.data.Count -ne $limit)
if ($locations) {
# Output the fetched locations to a CSV file
$locations | Export-Csv -Path "locations.csv" -NoTypeInformation
# Output success message with record count
$recordCount = $locations.Count
Write-Host "Successfully fetched and saved $recordCount records to locations.csv."
} else {
Write-Host "No Records were fetched."
}