apache-kafkalog4jksqldb

How to make ksql_processing_log and send error to kafka topic


I update log4j.properties in ksqldb and make auto create topic and stream in ksql.properties, but the error does not show in topic ksql_processing_log but shows in file ksql.log why?

and this my log4j.properties

log4j.rootLogger=INFO, stdout, ksqlAppender

log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=[%d] %p %m (%c)%n

# appenders
log4j.appender.ksqlAppender=org.apache.log4j.RollingFileAppender
log4j.appender.ksqlAppender.File={{ksql.appender_log_path}}{{ksql.appender_log_name}}
log4j.appender.ksqlAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.ksqlAppender.layout.ConversionPattern=[%d] %p %m (%c:%L)%n
log4j.appender.ksqlAppender.MaxFileSize={{ksql.appender_log_file_size}}
log4j.appender.ksqlAppender.MaxBackupIndex={{ksql.appender_max_log_files}}
log4j.appender.ksqlAppender.append=true

log4j.appender.streamsAppender=org.apache.log4j.RollingFileAppender
log4j.appender.streamsAppender.File={{ksql.appender_log_path}}{{ksql.streams_appender_log_name}}
log4j.appender.streamsAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.streamsAppender.layout.ConversionPattern=[%d] %p %m (%c:%L)%n
log4j.appender.streamsAppender.MaxFileSize={{ksql.streams_appender_log_file_size}}
log4j.appender.streamsAppender.MaxBackupIndex={{ksql.streams_appender_max_log_files}}
log4j.appender.streamsAppender.append=true

log4j.appender.kafkaAppender=org.apache.log4j.RollingFileAppender
log4j.appender.kafkaAppender.File={{ksql.appender_log_path}}{{ksql.kafka_appender_log_name}}
log4j.appender.kafkaAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.kafkaAppender.layout.ConversionPattern=[%d] %p %m (%c:%L)%n
log4j.appender.kafkaAppender.MaxFileSize={{ksql.kafka_appender_log_file_size}}
log4j.appender.kafkaAppender.MaxBackupIndex={{ksql.kafka_appender_max_log_files}}
log4j.appender.kafkaAppender.append=true

log4j.appender.kafka_appender=org.apache.kafka.log4jappender.KafkaLog4jAppender
log4j.appender.kafka_appender.layout=io.confluent.common.logging.log4j.StructuredJsonLayout
log4j.appender.kafka_appender.BrokerList=localhost:9092
log4j.appender.kafka_appender.Topic=default_ksql_processing_log
log4j.logger.processing=ERROR, kafka_appender

# loggers
log4j.logger.org.apache.kafka.streams=INFO, streamsAppender
log4j.additivity.org.apache.kafka.streams=false

log4j.logger.kafka=ERROR, kafkaAppender
log4j.additivity.kafka=false

log4j.logger.org.apache.zookeeper=ERROR, kafkaAppender
log4j.additivity.org.apache.zookeeper=false

log4j.logger.org.apache.kafka=ERROR, kafkaAppender
log4j.additivity.org.apache.kafka=false

log4j.logger.org.I0Itec.zkclient=ERROR, kafkaAppender
log4j.additivity.org.I0Itec.zkclient=false

log4j.logger.processing=ERROR, kafka_appender
log4j.additivity.processing=false

i dont know why log error does not produce to kafka topics


Solution

  • You need to add kafka_appender to the log4j.rootLogger line, if you want it to be used, by default, for all events.

    Otherwise, you need to use a correct KSQL class package name instead of processing

    For example,

    log4j.logger.io.confluent.ksql=ERROR, kafka_appender
    log4j.additivity.io.confluent.ksql=false