These are my settings
log.retention.check.interval.ms=300000 checks when logs can be deleted
log.retention.hours=168 how old a non active segment must be before it gets deleted
log.retention.ms=null how old a non active segment must be before it gets deleted
log.retention.bytes=-1 how big a non active segment must be before it gets deleted
segment.ms=30000 how old segments have to be before the log gets rotated.
from what I gather, a new log segment should be created every 30 seconds when possible(this is happening). But why are old segments deleted? Shouldn't it wait 168 hours(since log.retention.ms is null)?
there was a property I was not looking into "min.cleanable.dirty.ratio". After accounting for that, I was able to trigger compaction pretty regularly.