I am running Apache spark application in Yarn cluster. I use same application to trigger(spark-submit) multiple time with different parameters ( one of the parameter is --name ) and I have common log4j.xml file for logs.
I would like use some thing like ${spark.app.name} in log4j.xml to different logs from individual spark jobs.
I tried using ${spark.app.name} in log4j.xml but it is not working.
Tried :
I tried using ${spark.app.name} and ${spark.app.id} in log4j.xml
Expect to happen: Able to see spark job name in every line
What actually resulted: Spark job name didn't appeared in log instead it was empty.
Note : I have gone through this question Output Spark application name in driver log. It is not helping.
I Used separate log4j config file to name for every spark submit script.