pysparkdatabricksazure-databricksscddelta-live-tables

Delta live tables - Slowly changing dimensions


Is it possible to create an Slowly Changing Dimension mechanism using Delta Live Tables? I would like to implement something like this https://docs.databricks.com/_static/notebooks/merge-in-scd-type-2.html

But in the DLT docs i found "Processing updates from source tables, for example, merges and deletes, is not supported. To process updates, see the APPLY CHANGES INTO command" - but https://docs.databricks.com/data-engineering/delta-live-tables/delta-live-tables-cdc.html it's only for CDC, do you have any idea for SCD?


Solution

  • As you noticed right now DLT supports only SCD Type 1 (CDC). Support for SCD Type 2 is currently in the private preview, and should be available in near future - refer to the Databricks Q2 public roadmap for more details on it. If you have solutions architect or customer success engineer in your account, ask them to include you into private preview.

    Update, September 2022: SCD type 2 is generally available since June 2022: https://www.databricks.com/blog/2022/06/29/delta-live-tables-announces-new-capabilities-and-performance-optimizations.html