apache-sparkoptimizationdatabricksdelta-lakedelta

Can we perform DML and write operations when optimation of delta table is in progress?

In the documentation, they have mentioned that read operations are allowed during optimize operation, but nothing is mentioned about write and DML operations. If it is possible, can some one provide the link to the documention for same?

Solution

  • Delta format is designed to handle concurrent operations. While OPTIMIZE is running, other write and DML operations (like INSERT, UPDATE, DELETE, MERGE) can generally proceed. Delta's transactional guarantees ensure that these concurrent operations are handled correctly, maintaining data consistency.

    NOTE: However, heavy concurrent write operations during an optimize operation could increase the time it takes to complete the optimize operation.