I have two PostgreSQL tables: objects
and metadata
. Each object has a size
property that represents its size in bytes and can be stored in a unique location with storage_id
. In metadata, the total_size
of all objects of each storage with a given storage_id
is maintained. Simplified tables:
CREATE TABLE IF NOT EXISTS objects (
object_id UUID PRIMARY KEY,
storage_id UUID NOT NULL,
size BIGINT NOT NULL,
FOREIGN KEY (storage_id) REFERENCES metadata(storage_id)
);
CREATE TABLE IF NOT EXISTS metadata (
storage_id UUID PRIMARY KEY,
total_size BIGINT DEFAULT 0
);
To maintain the total_size
in the metadata
table, I have a triggers that whenever an object is inserted or deleted, the total_size
gets updated, i.e. here is the trigger for insertion:
CREATE OR REPLACE FUNCTION update_size_on_insert() RETURNS TRIGGER AS $$
BEGIN
UPDATE metadata
SET total_size = total_size + NEW.size
WHERE storage_id = NEW.storage_id;
RETURN NEW;
END;
$$ LANGUAGE plpgsql;
CREATE OR REPLACE TRIGGER trg_update_size_on_insert
AFTER INSERT ON objects
FOR EACH ROW
EXECUTE FUNCTION update_size_on_insert();
The problem is that concurrent insertions/deletions could override the total_size
which leads to invalid data. How can I change the trigger so it contains a row-level lock when the SET total_size = total_size + NEW.size
gets executed? I was thinking of the FOR UPDATE
statement, but that requires a SELECT
. Could I use PERFORM ... FOR UPDATE;
to lock the row?
Could I use
PERFORM ... FOR UPDATE;
to lock the row?
You could, but that just duplicates the lock your update
already acquires on its own, as immediately pointed out by @Iłya Bursov. You can see it in pg_locks
:
demo at db<>fiddle
start transaction;
create table t(v int primary key);
insert into t values(1),(2);
commit and chain;
update t set v=9 where v=1;
select mode
, relation::regclass
, pid=pg_backend_pid() as is_me
from pg_locks
join pg_database on pg_locks.database = pg_database.oid
and pg_database.datname = current_catalog
where relation::regclass in ('t','t_pkey');
mode | relation | is_me |
---|---|---|
RowExclusiveLock | t_pkey | t |
RowExclusiveLock | t | t |
One's on the table, the other on the index enforcing your primary key.
As hinted by @Frank Heikens, you might want to swap out the update
for an upsert, to also make sure your initial inserts are handled safely, and add a coalesce()
to cover any updates and deletes as recommended by @Charlieface:
CREATE OR REPLACE FUNCTION update_size_on_insert() RETURNS TRIGGER AS $$
BEGIN
INSERT INTO metadata AS m(storage_id,total_size)
VALUES(coalesce(NEW.storage_id,OLD.storage_id),coalesce(NEW.size,0))
ON CONFLICT(storage_id)DO UPDATE
SET total_size = m.total_size
+coalesce(NEW.size,0)
-coalesce(OLD.size,0);
RETURN NEW;
END $$ LANGUAGE plpgsql;
CREATE OR REPLACE TRIGGER trg_update_size_on_insert
AFTER INSERT OR UPDATE OR DELETE ON objects
FOR EACH ROW
EXECUTE FUNCTION update_size_on_insert();
To leave it as an after
trigger, you'll need to change the FK constraint. While you're at it, consider a CHECK
constraint to make sure nobody slips in negative-size objects:
size BIGINT NOT NULL CHECK(size>=0),
FOREIGN KEY (storage_id) REFERENCES metadata(storage_id) DEFERRABLE INITIALLY DEFERRED
Deferrability makes it give the trigger time to add the new storage_id
before the constraint gets validated. Without that, the trigger would have to fire before
.
insert into objects values
(gen_random_uuid(),gen_random_uuid(),1)
,(gen_random_uuid(),gen_random_uuid(),1)
returning*;
object_id | storage_id | size |
---|---|---|
35176f69-0f1f-4051-b903-9bb436c26b23 | 4d3d8298-0f2a-415f-b9a5-2784dafed066 | 1 |
bfda9791-2e11-4998-b98f-5b674b647a16 | 63d59ab2-de3a-442f-b05d-7708fb8d61dd | 1 |
Now metadata got populated on its own, automatically:
table metadata;
storage_id | total_size |
---|---|
4d3d8298-0f2a-415f-b9a5-2784dafed066 | 1 |
63d59ab2-de3a-442f-b05d-7708fb8d61dd | 1 |
Adding an object to existing storage:
insert into objects
select gen_random_uuid()
, storage_id
, 9 as size
from objects
limit 1 --any existing storage will do
returning*;
object_id | storage_id | size |
---|---|---|
b60b5742-45e0-4118-97ab-fd60b915e9c2 | 4d3d8298-0f2a-415f-b9a5-2784dafed066 | 9 |
table metadata;
storage_id | total_size |
---|---|
63d59ab2-de3a-442f-b05d-7708fb8d61dd | 1 |
4d3d8298-0f2a-415f-b9a5-2784dafed066 | 10 new object increased the total size |
Updating the size of any one arbitrary object:
update objects
set size=5
where ctid=(select min(ctid)from objects)
returning*;
object_id | storage_id | size |
---|---|---|
35176f69-0f1f-4051-b903-9bb436c26b23 | 4d3d8298-0f2a-415f-b9a5-2784dafed066 | 5 |
table metadata;
storage_id | total_size |
---|---|
63d59ab2-de3a-442f-b05d-7708fb8d61dd | 1 |
4d3d8298-0f2a-415f-b9a5-2784dafed066 | 14 |
delete from objects
where ctid=(select min(ctid)from objects)
returning*;
object_id | storage_id | size |
---|---|---|
bfda9791-2e11-4998-b98f-5b674b647a16 | 63d59ab2-de3a-442f-b05d-7708fb8d61dd | 1 |
table metadata;
storage_id | total_size |
---|---|
4d3d8298-0f2a-415f-b9a5-2784dafed066 | 14 |
63d59ab2-de3a-442f-b05d-7708fb8d61dd | 0 |
Optionally, you can expand the trigger to also take care of removing totals for unused storage where total_size
dropped to zero.
To handle a truncate
event, you also need a separate statement level trigger - even though that deletes all rows of a table, it does not fire on delete
triggers.
@Adrian Klaver's view
is good thing to consider:
materialized view
and cache them periodically.