typescriptpostgresqlsortingpaginationkeyset-pagination

Key-set pagination order/filter on timestamp that does not have the same precision


I'm encountering an issue with my Key-set Pagination functionality because the timestamp value I receive is constrained to milliseconds, causing a mismatch with the precision expected by the database. This limitation stems from the inherent lack of precision in the JavaScript Date implementation. Unfortunately, I have no control over this aspect as the value is obtained from an API endpoint. While one workaround involves transmitting a bigint string value to and from the client, I'm interested in exploring alternative solutions to this problem before resorting to potentially time-consuming changes.

Schema (PostgreSQL v15)

CREATE TABLE IF NOT EXISTS public.activities
(
    id uuid NOT NULL DEFAULT gen_random_uuid(),    
    created_at timestamp with time zone DEFAULT now(),
    PRIMARY KEY (id)
);

CREATE INDEX activities_created_at_id ON activities (created_at, id);

Ordering

Below is the table's content sorted by created_at asc, id asc

select id, created_at from activities order by created_at asc, id asc;
id created_at
e34d5557-43d7-4f81-802b-791c213ea5cb 2024-03-14T07:23:56.474123Z
ea7e9cad-89f2-4610-898f-62a04e8b5331 2024-03-14T07:23:56.474123Z
0cf727a8-5efc-454b-ba1b-ea301e2b1a82 2024-03-14T07:23:56.474222Z
10f1dd9b-2cd8-40bb-a199-2d9c922d07b1 2024-03-14T07:23:56.474222Z
3c00c38e-45b9-4c3c-9d05-fc1ca2177dbe 2024-03-14T07:23:56.474222Z
ac31e591-d4ea-4ef5-956a-c07bd043d9ea 2024-03-14T07:23:56.474222Z
b9b33ca5-2cd1-490b-a6be-d28784f75b2a 2024-03-14T07:23:56.474222Z
bc9763d0-6d24-4fbf-bc23-7eff1589280a 2024-03-14T07:23:56.474222Z
c442f6ea-61bf-42d4-aa85-4eab1d95f569 2024-03-14T07:23:56.474222Z
ded826f7-65f0-4d23-9366-049823ba49ec 2024-03-14T07:23:56.474222Z

Issue with Significant Digits

The input provided lacks precision compared to what's stored in the database. Due to JavaScript's limitation in supporting precision beyond milliseconds, the following query retrieves all activities:

select id, created_at from activities
where (created_at, id) > ('2024-03-14T07:23:56.474Z', 'b9b33ca5-2cd1-490b-a6be-d28784f75b2a')
order by created_at asc, id asc;
id created_at
e34d5557-43d7-4f81-802b-791c213ea5cb 2024-03-14T07:23:56.474Z
ea7e9cad-89f2-4610-898f-62a04e8b5331 2024-03-14T07:23:56.474Z
0cf727a8-5efc-454b-ba1b-ea301e2b1a82 2024-03-14T07:23:56.474Z
10f1dd9b-2cd8-40bb-a199-2d9c922d07b1 2024-03-14T07:23:56.474Z
3c00c38e-45b9-4c3c-9d05-fc1ca2177dbe 2024-03-14T07:23:56.474Z
ac31e591-d4ea-4ef5-956a-c07bd043d9ea 2024-03-14T07:23:56.474Z
b9b33ca5-2cd1-490b-a6be-d28784f75b2a 2024-03-14T07:23:56.474Z
bc9763d0-6d24-4fbf-bc23-7eff1589280a 2024-03-14T07:23:56.474Z
c442f6ea-61bf-42d4-aa85-4eab1d95f569 2024-03-14T07:23:56.474Z
ded826f7-65f0-4d23-9366-049823ba49ec 2024-03-14T07:23:56.474Z

Using date_trunc to Cap Precision When Querying

While achieving the desired result, I'm hesitant to handle this solely within the query. This approach would necessitate dynamically adjusting the query construction based on the field type, potentially introducing performance concerns:

select id, created_at from activities
where (date_trunc('milliseconds', created_at::timestamptz), id) < (date_trunc('milliseconds', '2024-03-14T07:23:56.474Z'::timestamptz), 'b9b33ca5-2cd1-490b-a6be-d28784f75b2a')
order by created_at desc, id desc;
id created_at
ac31e591-d4ea-4ef5-956a-c07bd043d9ea 2024-03-14T07:23:56.474Z
3c00c38e-45b9-4c3c-9d05-fc1ca2177dbe 2024-03-14T07:23:56.474Z
10f1dd9b-2cd8-40bb-a199-2d9c922d07b1 2024-03-14T07:23:56.474Z
0cf727a8-5efc-454b-ba1b-ea301e2b1a82 2024-03-14T07:23:56.474Z

Considering these options, I'm open to suggestions and insights from the community to address this pagination challenge effectively.

View on DB Fiddle


Solution

  • If PostgreSQL microsecond resolution just gets in the way, you don't need to use it: it's customisable.

    time, timestamp, and interval accept an optional precision value p which specifies the number of fractional digits retained in the seconds field. By default, there is no explicit bound on precision. The allowed range of p is from 0 to 6.

    You can alter the column type in-place, which will simply round the last 3 digits in the field, in all existing and future rows in the table: demo

    alter table activities alter column created_at type timestamptz(3);
    
    select id, 
           to_char(created_at,
                   'YYYY-MM-DD"T"HH24:MI:SS.USTZ'),
           to_char('2024-03-14T07:23:56.474Z'::timestamptz,
                   'YYYY-MM-DD"T"HH24:MI:SS.USTZ') target
    from activities
    where (created_at, id) 
        > ('2024-03-14T07:23:56.474Z', 'b9b33ca5-2cd1-490b-a6be-d28784f75b2a')
    order by created_at asc, id asc;
    
    id to_char target
    bc9763d0-6d24-4fbf-bc23-7eff1589280a 2024-03-14T07:23:56.474000UTC 2024-03-14T07:23:56.474000UTC
    c442f6ea-61bf-42d4-aa85-4eab1d95f569 2024-03-14T07:23:56.474000UTC 2024-03-14T07:23:56.474000UTC
    ded826f7-65f0-4d23-9366-049823ba49ec 2024-03-14T07:23:56.474000UTC 2024-03-14T07:23:56.474000UTC
    e34d5557-43d7-4f81-802b-791c213ea5cb 2024-03-14T07:23:56.474000UTC 2024-03-14T07:23:56.474000UTC
    ea7e9cad-89f2-4610-898f-62a04e8b5331 2024-03-14T07:23:56.474000UTC 2024-03-14T07:23:56.474000UTC

    The lower resolution column can also be generated based on the original one, in case the microseconds are or might at one point become useful elsewhere:

    alter table activities 
      add column created_at_mili_precise timestamptz(3) 
        generated always as (created_at::timestamptz(3)) stored;