I'm encountering an issue with my Key-set Pagination functionality because the timestamp value I receive is constrained to milliseconds, causing a mismatch with the precision expected by the database. This limitation stems from the inherent lack of precision in the JavaScript Date implementation. Unfortunately, I have no control over this aspect as the value is obtained from an API endpoint. While one workaround involves transmitting a bigint string value to and from the client, I'm interested in exploring alternative solutions to this problem before resorting to potentially time-consuming changes.
Schema (PostgreSQL v15)
CREATE TABLE IF NOT EXISTS public.activities
(
id uuid NOT NULL DEFAULT gen_random_uuid(),
created_at timestamp with time zone DEFAULT now(),
PRIMARY KEY (id)
);
CREATE INDEX activities_created_at_id ON activities (created_at, id);
Ordering
Below is the table's content sorted by created_at asc
, id asc
select id, created_at from activities order by created_at asc, id asc;
id | created_at |
---|---|
e34d5557-43d7-4f81-802b-791c213ea5cb | 2024-03-14T07:23:56.474123Z |
ea7e9cad-89f2-4610-898f-62a04e8b5331 | 2024-03-14T07:23:56.474123Z |
0cf727a8-5efc-454b-ba1b-ea301e2b1a82 | 2024-03-14T07:23:56.474222Z |
10f1dd9b-2cd8-40bb-a199-2d9c922d07b1 | 2024-03-14T07:23:56.474222Z |
3c00c38e-45b9-4c3c-9d05-fc1ca2177dbe | 2024-03-14T07:23:56.474222Z |
ac31e591-d4ea-4ef5-956a-c07bd043d9ea | 2024-03-14T07:23:56.474222Z |
b9b33ca5-2cd1-490b-a6be-d28784f75b2a | 2024-03-14T07:23:56.474222Z |
bc9763d0-6d24-4fbf-bc23-7eff1589280a | 2024-03-14T07:23:56.474222Z |
c442f6ea-61bf-42d4-aa85-4eab1d95f569 | 2024-03-14T07:23:56.474222Z |
ded826f7-65f0-4d23-9366-049823ba49ec | 2024-03-14T07:23:56.474222Z |
Issue with Significant Digits
The input provided lacks precision compared to what's stored in the database. Due to JavaScript's limitation in supporting precision beyond milliseconds, the following query retrieves all activities:
select id, created_at from activities
where (created_at, id) > ('2024-03-14T07:23:56.474Z', 'b9b33ca5-2cd1-490b-a6be-d28784f75b2a')
order by created_at asc, id asc;
id | created_at |
---|---|
e34d5557-43d7-4f81-802b-791c213ea5cb | 2024-03-14T07:23:56.474Z |
ea7e9cad-89f2-4610-898f-62a04e8b5331 | 2024-03-14T07:23:56.474Z |
0cf727a8-5efc-454b-ba1b-ea301e2b1a82 | 2024-03-14T07:23:56.474Z |
10f1dd9b-2cd8-40bb-a199-2d9c922d07b1 | 2024-03-14T07:23:56.474Z |
3c00c38e-45b9-4c3c-9d05-fc1ca2177dbe | 2024-03-14T07:23:56.474Z |
ac31e591-d4ea-4ef5-956a-c07bd043d9ea | 2024-03-14T07:23:56.474Z |
b9b33ca5-2cd1-490b-a6be-d28784f75b2a | 2024-03-14T07:23:56.474Z |
bc9763d0-6d24-4fbf-bc23-7eff1589280a | 2024-03-14T07:23:56.474Z |
c442f6ea-61bf-42d4-aa85-4eab1d95f569 | 2024-03-14T07:23:56.474Z |
ded826f7-65f0-4d23-9366-049823ba49ec | 2024-03-14T07:23:56.474Z |
Using date_trunc
to Cap Precision When Querying
While achieving the desired result, I'm hesitant to handle this solely within the query. This approach would necessitate dynamically adjusting the query construction based on the field type, potentially introducing performance concerns:
select id, created_at from activities
where (date_trunc('milliseconds', created_at::timestamptz), id) < (date_trunc('milliseconds', '2024-03-14T07:23:56.474Z'::timestamptz), 'b9b33ca5-2cd1-490b-a6be-d28784f75b2a')
order by created_at desc, id desc;
id | created_at |
---|---|
ac31e591-d4ea-4ef5-956a-c07bd043d9ea | 2024-03-14T07:23:56.474Z |
3c00c38e-45b9-4c3c-9d05-fc1ca2177dbe | 2024-03-14T07:23:56.474Z |
10f1dd9b-2cd8-40bb-a199-2d9c922d07b1 | 2024-03-14T07:23:56.474Z |
0cf727a8-5efc-454b-ba1b-ea301e2b1a82 | 2024-03-14T07:23:56.474Z |
Considering these options, I'm open to suggestions and insights from the community to address this pagination challenge effectively.
If PostgreSQL microsecond resolution just gets in the way, you don't need to use it: it's customisable.
time
,timestamp
, andinterval
accept an optional precision value p which specifies the number of fractional digits retained in the seconds field. By default, there is no explicit bound on precision. The allowed range of p is from 0 to 6.
You can alter
the column type in-place, which will simply round the last 3 digits in the field, in all existing and future rows in the table: demo
alter table activities alter column created_at type timestamptz(3);
select id,
to_char(created_at,
'YYYY-MM-DD"T"HH24:MI:SS.USTZ'),
to_char('2024-03-14T07:23:56.474Z'::timestamptz,
'YYYY-MM-DD"T"HH24:MI:SS.USTZ') target
from activities
where (created_at, id)
> ('2024-03-14T07:23:56.474Z', 'b9b33ca5-2cd1-490b-a6be-d28784f75b2a')
order by created_at asc, id asc;
id | to_char | target |
---|---|---|
bc9763d0-6d24-4fbf-bc23-7eff1589280a | 2024-03-14T07:23:56.474000UTC | 2024-03-14T07:23:56.474000UTC |
c442f6ea-61bf-42d4-aa85-4eab1d95f569 | 2024-03-14T07:23:56.474000UTC | 2024-03-14T07:23:56.474000UTC |
ded826f7-65f0-4d23-9366-049823ba49ec | 2024-03-14T07:23:56.474000UTC | 2024-03-14T07:23:56.474000UTC |
e34d5557-43d7-4f81-802b-791c213ea5cb | 2024-03-14T07:23:56.474000UTC | 2024-03-14T07:23:56.474000UTC |
ea7e9cad-89f2-4610-898f-62a04e8b5331 | 2024-03-14T07:23:56.474000UTC | 2024-03-14T07:23:56.474000UTC |
The lower resolution column can also be generated based on the original one, in case the microseconds are or might at one point become useful elsewhere:
alter table activities
add column created_at_mili_precise timestamptz(3)
generated always as (created_at::timestamptz(3)) stored;