apache-kafka

Kafka compact cleanup policy and config that could cleanup if segment is to big

We have a kafka topic set with a cleanup.policy of compact. we currently have the segment.bytes set to 1GB.

I have read the documentation but it wasnt 100% clear so I thought I could ask here.

Due to there being lots of config values, will any of these effect a compact policy, in that when a segment fills up, would it ever be eligible for deletion and hence we would loose the old data?

Just wondering what the right setup to keep them forever when the policy is compact.

Below shows our current config values:

enter image description here enter image description here

Solution

  • If you want compaction and deletion, you need to set cleanup.policy=compact,delete

    Only then would retention.ms, retention.bytes or similar configs be applied.