Suppose I have a table with three columns A, B, C. if source stops sending B data (Which we assume that it got deleted in file, Not in table schema). Few days later, if source wants to add that column back but in another position like B, A, C. How iceberg tackles this situation?? How will iceberg map the column names? I tried to use some iceberg table properties but nothing worked. It's throwing an error
In Iceberg, column mapping is handled through column IDs rather than relying on column positions or names.
1. Initial Table Schema:
Columns: A, B, C Each column gets assigned a unique column ID by Iceberg.
2. When Column B is Missing:
If the source stops sending B but the schema still includes B, Iceberg will interpret this as null values for B in new data files and No changes are made to the schema.
3. Adding Back B:
When the source starts sending back B but in a new position (e.g., B, A, C), the schema evolution will fail if you try to map B as a new column or reorder columns without explicitly reconciling. This is obviously due to the fact that, Iceberg identifies columns by their IDs and not by their positions. And If you attempt to re-add B without ensuring it matches the original column ID, Iceberg treats it as a new column, leading to a conflict.
So what's the solution?
Ensure Column Mapping is Consistent:
To restore B correctly, ensure that its column ID matches its original ID. This requires updating the schema to reintroduce B using its existing metadata. The idea is to REORDER instead of just Adding/Dropping the column.
ALTER TABLE table_name REPLACE COLUMN `B` COMMENT 'Restoring/Reordering B to new position';
Best way to do this is -
NOTE : By default, Iceberg maintains strict schema evolution rules. You can adjust properties like schema.alter.order.columns to enable column reordering. However, this doesn’t affect column ID mapping and does not solve this issue directly.
Best Practices:
column IDs rather than column positions or column names for mapping.