apache-sparkairflowlivy

how to switch Livy job log-level into DEBUG mode?


Like in spark we can add -Dlog4j.debug config , do we have any such equivalent in Airflow Livy operator. I have already browsed https://airflow.apache.org/docs/apache-airflow-providers-apache-livy/stable/_api/airflow/providers/apache/livy/operators/livy/index.html for the same.


Solution

  • The livy operator requires a running livy server. You'd pass log4j arguments when you start the livy JVM process; you can't set them from a Python airflow process

    If you mean, pass to the Spark driver/executor, then yes, that's just part of the SparkConf / SparkSession options, set in the conf argument, or args, depending on your own app.