databasepostgresqldatabase-replicationdebezium

How to change the confirmed_flush_lsn to a smaller value


Problems

Currently, when creating a replication slot using CREATE_REPLICATION_SLOT, the values of catalog_xmin, restart_lsn, and confirmed_flush_lsn are set to the latest xid and lsn.

However, during the subsequent consumption of WAL log data, the server is instructed to start streaming WAL for logical replication, starting at either WAL location XXX/XXX or the slot's confirmed_flush_lsn (see Section 54.19), whichever is greater. This means that it is not possible to consume data from before the confirmed_flush_lsn.

If I want to start consuming from an earlier position in the streaming process, how to change the confirmed_flush_lsn to a smaller value?( Obviously, pg_slot_advance can’t do that)

How to change the confirmed_flush_lsn to a smaller value? or how to initialize an specified earlier confirmed_flush_lsn slot?

Situations

For example, in Flink CDC or《DBLog: A Generic Change-Data-Capture Framework》, read snapshot in multiple chunks in parallel. For each chunks:

  1. read current Lsn as low_watermark.
  2. read snapshot chunk.
  3. read current Lsn as high_watermark.
  4. backfill logs between [low_watermark, high_watermark).

To decease slot numbers pressure, I want to read low_watermark and high_watermark without slot creation. Create this slot later when need. (Meantime, a global slot is retained to avoid log to be recycled.

However, a slot create later cannot read Wal-Log before created.


Solution

  • Just by using built-in commands or utilities, you cannot change the confirmed_flush_lsn in PostgreSQL to a smaller value or initialize it to an earlier position once a replication slot has been created.The confirmed_flush_lsn represents the point up to which the data has been safely replicated, and it's not intended to be moved backward.

    Usually, you would need to establish a new replication slot and begin replication from the desired point if you want to consume data from an earlier position in the WAL stream.