azure-data-factoryazure-cosmosdbazure-synapse-analyticsazure-cosmosdb-changefeed

How to Access "Checkpoint Key" Property in Data Flow Activity Settings in Azure Synapse Analytics and Data Factory


Context I am currently using Azure Synapse Analytics to extract change feed data from CosmosDB. My setup involves:

Issue: I need to override the checkpoint key in the pipeline. However, I am encountering a challenge:

Question: What are the prerequisites or specific conditions that trigger the visibility of the "Checkpoint Key" property in the settings tab of a data flow activity in both Azure Synapse Analytics and Azure Data Factory?

Attempts to Resolve: I have recreated the pipelines and dataflows in Data Factory in order to determine if it was an issue with my Synapse configuration with no success. The issue exists in both Synapse and ADF.

Expected Outcome: I am looking to understand how to access or enable the "Checkpoint Key" property in the data flow activity settings to override it in my pipeline.


Solution

  • The Checkpoint Key shows only when you enable Change data capture in the source configuration of a dataflow activity. I have checked with DelimitedText as a source, and it was working fine.

    However, when I selected the source as CosmosDb, by default Change data capture was enabled, but I still faced the same issue: the Checkpoint Key was not showing.

    enter image description here

    The Checkpoint Key will be displayed according to the dataset selected.

    So, I figured out that I first needed to enable Change data capture by selecting DelimitedText as the input type in Source settings.

    enter image description here

    and

    enter image description here

    Then, you will get the Checkpoint Key.

    enter image description here

    After that, you can change the source data to CosmosDb and will be able to Override.

    enter image description here