A solution I am working on requires that I request 100k plus data rows from Athena in a spring api. I am led to believe that only 1000 rows at a time are provided by default. Wondering if anyone has experience with a similar operation and how to request large datasets from Athena without paging
After the query is done you can use the get_query_execution function to find the output location and download it as a CSV file. If the output is very large it is possible to use an UNLOAD or a CTAS query, in which you can control the output format and the output location. I used it to save the data as json.gz. Later I downloaded the output files and iterated the records locally.