apache-flinkupsertflink-sql

Where the Upsert Kafka connector consumer start?

I'm trying to read from kafka in upsert mode. Everything works fine. But I noticed that scan.startup.mode is not supported.

Is my assumption right that in upsert mode kafka connector reads the whole topic from earliest offset by design or there is another logic? If I need to reduce amount of processed data should I reduce retention of topic?

Flink version: 1.14.6

Solution

  • Reading from the beginning is necessary for correctness. You can compact the topic to reduce the effort and storage required.