apache-sparkkuberneteslogginglog4j

Spark Executor STDOUT to Kubernetes STDOUT

My spark application running inside Spark Worker outputs the executor logs to a specific file path: "/worker_home_directory/app-xxxxxxxx/0/stdout"

I used log4j.properties to redirect logs from stderr to stdout, but it seems like these are still not recognized by Kubernetes as the main STDOUT logs.

My goal is to use kubectl logs <spark_worker_pod> and view the Spark executor logs directly. However, my current solution seems half-baked.

How do I move the logs from /worker_home_directory/app-xxxxxxxx/0/stdout to Kubernetes STDOUT?

Solution

  • You need to set your log4j properties to send logs to STDOUT

    Something along the lines as below:

    log4j.rootLogger=ERROR, stdout

log4j.appender.STDOUT=org.apache.log4j.ConsoleAppender