[SOLVED] Kafka Best Practices + how to set recommended setting for JVM

Kafka Best Practices + how to set recommended setting for JVM

A recommended setting for JVM looks like following

-Xmx8g -Xms8g -XX:MetaspaceSize=96m -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M -XX:MinMetaspaceFreeRatio=50 -XX:MaxMetaspaceFreeRatio=80

my question is - How do I set the above Java options for Kafka?

I know for sure that we can set

export KAFKA_HEAP_OPTS="-Xmx8G -Xms8G"

but not sure if we can append the whole line

"-Xmx8g -Xms8g -XX:MetaspaceSize=96m -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M -XX:MinMetaspaceFreeRatio=50 -XX:MaxMetaspaceFreeRatio=80"

to KAFKA_HEAP_OPTS variable

reference - https://community.hortonworks.com/articles/80813/kafka-best-practices-1.html

Solution

you can check kafka-run-class.sh, here you can see what env variables kafka uses to start the java process:

$KAFKA_HEAP_OPTS
$KAFKA_JVM_PERFORMANCE_OPTS
$KAFKA_GC_LOG_OPTS
$KAFKA_JMX_OPTS
$KAFKA_LOG4J_OPTS

and then it run the java application passing them:

nohup $JAVA $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS....

basically the content of the env variables are just appended as is to the command, it does not really matter where you put the settings as soon as the jvm parameters are in the correct order.

So, you can simply change $KAFKA_HEAP_OPTS however to keep the variable names consistent with their content I would put -Xmx8g -Xms8g in KAFKA_HEAP_OPTS and the remaining optimization to KAFKA_JVM_PERFORMANCE_OPTS.