postgresqldatabase-backupswal

Match Postgres WAL Segments to According Base Backup


As both the archive_command and restore_command defined with related config options looks to be injected only with %f and %p parameter how actually to identify on top of what base backup the current segment is archived/restored on.

So that having an external storage for the backup artifacts how for the scenarios similar to

initdb
   switch_wal
   switch_wal
   switch_wal
pg_basebackup
   switch_wal
   switch_wal
pg_basebackup
   switch_wal
   switch_wal
   switch_wal

to be able to manage related artifacts (e.g. copy/remove/cleanup) in logically atomic way as

initdb
   switch_wal
   switch_wal
   switch_wal
pg_basebackup
   switch_wal
   switch_wal
pg_basebackup
   switch_wal
   switch_wal
   switch_wal

Is maybe the prefix in WAL segment names (eg 000000010000000000000005, 00000008000000000000000A) the one that could help with this

Updated

As I understand the WAL segments are kind of incremental diffs that could be applied on top of specific database state. As @Laurenz Albe mentioned in the comment it doesn’t belong to a specific base backup. Still I assume they doesn’t have any sens in terms of any other (non related base backup) either. I.e. you can't apply WAL segment to a state the database was restored from any other non related base backup. Or can you?

So my intention is to organize an external storage for the archives in logical groups chronologically organized around the base backup points. E.g.

/base-backup-point-1
   backup-tar
   wal-segment-1
   wal-segment-2
   wal-segment-3

/base-backup-point-2
   backup-tar
   wal-segment-4
   wal-segment-5

/base-backup-point-3
   backup-tar
   wal-segment-1
   wal-segment-2
   wal-segment-3
   wal-segment-4

so that


Solution

  • Every base backup will archive a *.backup file in the WAL archive that contains the earliest WAL segment that is required to recover that backup. The same information can be found in the backup.label file that is part of every base backup. So that part is simple.

    However, an archived WAL segment does not belong to a single base backup. Rather, you need all WAL segments after the initial one.

    If you want to purge the WAL archive, you can use pg_archivecleanup. As arguments, pass the WAL archive directory and the oldest *.backup file you want to retain. The utility will delete all older files.