I am running 3 nodes Kafka cluster on Windows Server 2019. I have set the default retention time to 72 hours. All was working fine for some days but once in a while, I am getting the following exception which causes the Kafka Broker shutting down.
Line 3607: [2020-03-15 07:45:24,047] ERROR Error while deleting segments for kafka-monitor-topic-0 in dir E:\uhx\kafka\kafka-logs (kafka.server.LogDirFailureChannel)
Line 3653: [2020-03-15 07:45:24,047] ERROR Uncaught exception in scheduled task 'kafka-log-retention' (kafka.utils.KafkaScheduler)
Line 3707: [2020-03-15 07:45:24,094] ERROR Shutdown broker because all log dirs in E:\kafka\kafka-logs have failed (kafka.log.LogManager)
Line 4187: [2020-03-15 07:45:37,149] ERROR [Log partition=test-0, dir=E:\uhx\kafka\kafka-logs] Could not find offset index file corresponding to log file E:\kafka\kafka-logs\test-0\00000000000000000001.log, recovering segment and rebuilding index files... (kafka.log.Log)
java.nio.file.FileSystemException: E:\kafka\kafka-logs\test-2\00000000000000000000.index -> E:\kafka\kafka-logs\test-2\00000000000000000000.index.deleted: The process cannot access the file because it is being used by another process.
When I searched for it, I got to know it is already in the Open Jira issues of Kafka https://issues.apache.org/jira/browse/KAFKA-1194 But now I need to know if there is any workaround to handle this situation and also clear the logs without the retention policy of Kafka.
One workaround would be to run Kafka in WSL2 as a Linux process
Other solutions include Docker or buying new servers
Fact of the matter is that there's no workaround for Windows file management until the ticket is addressed