amazon-web-servicesamazon-qldb

How to enable paging for AWS QLDB search query results in python?

I am currently designing a project where my AWS Lambda Function in Python needs to run a search query on a table from a QLDB Ledger. The lambda function implements an API gateway endpoint which is invoked from a HTML web UI. Therefore, I need to enable paging for the search query results. How to make that happen in python?

Solution

  • Addressed Dan's comments.

    High level design that you have :

    Web UI(Search) -> API Gateway -> AWS Lambda -> QLDB

    [Clarification] Is your requirement that Web UI would like to show one page of the search result from QLDB and then show the next page based on your customer's action ?

    1. If no, can you please elaborate your use case.
    2. If yes, queries to QLDB are bounded by transaction. A transaction can have a lifespan of 30 seconds. We do not support pagination across transactions. QLDB also in current form does not support LIMIT or OFFSET.

    Assuming you intended #2,

    Looks like you are trying to implement search capability on top of QLDB. While QLDB can support basic query capabilities its primarily tuned to support high throughput transactional writes. A search query on QLDB on an un-indexed column can lead to full table scans, which can be highly inefficient for large tables. Streaming QLDB data to Aurora or ElasticSearch and supporting search query on top of it, might be a better option. Here are some helpful links :

    Sample application showing integration with Aurora : https://qldb-immersionday.workshop.aws/en/lab4.html Sample application showing integration with ElasticSearch : https://docs.aws.amazon.com/qldb/latest/developerguide/streams.sdk.html#streams.sdk.samples.elastic

    Developer guide links which talks about pattern's and anti-patterns :