apache-sparkhadoopapache-iceberg

Cannot expire snapshot with retain last properies


I have 67 snapshot in a single table but when i use CALL

iceberg_catalog.system.expire_snapshots(
table => 'iceberg_catalog.default.test_7',
retain_last => 5
);

It doesn't delete any snapshot. My table use 2 properties which is :

write.metadata.delete-after-commit.enabled true write.metadata.previous-versions-max 5

can someone explain, tks very much


Solution

  • As per the documents, data files are not deleted until there are references by some active snapshot.
    https://iceberg.apache.org/docs/latest/spark-procedures/#usage_7
    https://iceberg.apache.org/docs/1.5.1/maintenance/#expire-snapshots

    If older_than and retain_last are omitted, the table's expiration properties will be used. Snapshots that are still referenced by branches or tags won't be removed. By default, branches and tags never expire, but their retention policy can be changed with the table property history.expire.max-ref-age-ms. The main branch never expires.

    Data files are not deleted until they are no longer referenced by a snapshot that may be used for time travel or rollback. Regularly expiring snapshots deletes unused data files.

    With the above two references , either history.expire.max-ref-age-ms setting to expire tags and other metadata or the references by some active snapshot seems to be the reason.