Kafka compact cleanup policy and config that could cleanup if segment is to big
We have a kafka topic set with a cleanup.policy of compact.
we currently have the segment.bytes set to 1GB.
I have read the documentation but it wasnt 100% clear so I thought I could ask here.
Due to there being lots of config values, will any of these effect a compact policy, in that when a segment fills up, would it ever be eligible for deletion and hence we would loose the old data?
Just wondering what the right setup to keep them forever when the policy is compact.
Below shows our current config values:
Solution
If you want compaction and deletion, you need to set cleanup.policy=compact,delete
Only then would retention.ms, retention.bytes or similar configs be applied.
- In docker-compose, my init-kafka image is failing to list topics on my kafka image. How do I fix this? (With KRaft, not Zookeeper!)
- Unknown Magic Byte - Kafka Json
- Monitor the lag for the consumers that are assigned to partitions topic
- Kafka compact cleanup policy and config that could cleanup if segment is to big
- Connect to Kafka running in Docker
- How to publish messages to Kafka from another container
- what message delivery guarantees to choose for KAFKA CQRS
- How to pass multiple bootstrap servers for listener using spring-kafka
- Kafka + Java app. in Docker on local computer
- Storing chat messages for a messaging microservice
- Kafka : ClassCastException: class org.apache.avro.generic.GenericData$Record cannot be cast to class
- Spring boot 3.2.9 Cloud stream - Kafka binder. classcastexception - class java.lang.Byte cannot be cast to class java.util.UUID
- Spring Boot Kafka Micrometer observability doesn't work with concurrency enabled
- Kafka Streams Admin Client disconnections
- How to disable a method annotated with @KafkaListener runtime
- Kafka topic not created in KafkaContainer
- Does spring always require KafkaTemplate?
- Connection to node -1 (/127.0.0.1:9092) could not be established. Broker may not be available
- Docker compose doesn't save data in volume
- Cmake not found
- How do I access the fields within a VARIANT column while reading from Kafka using Spark?
- Launching multiple Kafka brokers fails - Configured brokerId doesn't match stored brokerId
- How to make generic kafka consumer factory?
- Spring Kafka The class is not in the trusted packages
- How can I access Confluent Cloud Schema Registry using KafkaTopical?
- Kafka Best Way to Filter Messages
- Healthcheck not working at all when using docker-compose (My service do not wait for Kafka to be started before launching)
- Error to serialize message when sending to kafka topic
- How to write spark streaming DF to Kafka topic
- ERROR Uncaught exception in REST call to /metrics (org.apache.kafka.connect.runtime.rest.errors.ConnectExceptionMapper:61)