I have a 3-node systemd kafka cluster. I had removed all log files in /opt/kafka/kafka/logs/ in 1 node, thought they were just storage-consuming logs. Now after restarting the zookeeper.service I get the following error form systemctl status zookeeper.service which makes it unable to start:
systemd[1]: Started zookeeper.service.
zookeeper-server-start.sh[23029]: [0.001s][error][logging] Error opening log file '/opt/kafka/kafka/bin/../logs/zookeeper-gc.log': Permission denied
zookeeper-server-start.sh[23029]: [0.002s][error][logging] Initialization of output 'file=/opt/kafka/kafka/bin/../logs/zookeeper-gc.log' using options 'filecount=10,filesize=100M' failed.
zookeeper-server-start.sh[23029]: Invalid -Xlog option '-Xlog:gc*:file=/opt/kafka/kafka/bin/../logs/zookeeper-gc.log:time,tags:filecount=10,filesize=100M', see error log for details.
zookeeper-server-start.sh[23029]: Error: Could not create the Java Virtual Machine.
zookeeper-server-start.sh[23029]: Error: A fatal exception has occurred. Program will exit.
systemd[1]: zookeeper.service: Main process exited, code=exited, status=1/FAILURE
systemd[1]: zookeeper.service: Failed with result 'exit-code'.
And therefore kafka.service can't be started since it requires zookeeper.service.
I have tried to create the zookeeper-gc.log manually and configured the owner, group, and permissions based on other nodes' same log file:
touch /opt/kafka/kafka/bin/../logs/zookeeper-gc.log
chown kafka:kafka /opt/kafka/kafka/bin/../logs/zookeeper-gc.log
chmod 644 /opt/kafka/kafka/bin/../logs/zookeeper-gc.log
And restarted both services but got the same output.
I also find the following lines in /opt/kafka/kafka/bin/kafka-run-class.sh:
KAFKA_GC_LOG_OPTS="-Xlog:gc*:file=$LOG_DIR/$GC_LOG_FILE_NAME:time,tags:filecount=10,filesize=100M"
KAFKA_GC_LOG_OPTS="-Xloggc:$LOG_DIR/$GC_LOG_FILE_NAME -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSi
But had not enogh knowledge to modify or change them. Any help would be appreciate.
From the error message of your systemctl restart kafka zookeeper command's output, it seems either permissions or ownership of either the /opt/kafka/kafka/bin/../logs/zookeeper-gc.log file or the /opt/kafka/kafka/bin/../logs/ directory is not correct.
I don't know which OS you're running, but try the following and see if it fixes the problem:
chown -R kafka:kafka /opt/kafka/kafka/bin/../logs/
Note: The -R flag means recursive.