I'm using Spark 2.1.1 . This is my problem:
I have 2 files in the same directory named tools. One is main_process.scala and the other one is main_process_fun.scala. The files, basically looks like this:
1.- main_process.scala:
package mod.pack.tools
import x.y.spark.InitSpark
import com.typesafe.config.Config
trait main_process extends InitSpark with main_process_fun {
this: InitSpark =>
/**
* @param spark Initialized SparkSession
* @param config Config retrieved from args
*/
override def run(spark: SparkSession, config: Config): Int = {
logger.info("LOG")
implicit val sparkSes = spark
pre_defined_f1(config).flatMap(cfg => pre_defined_f2(cfg).flatMap(pre_defined_f3(_, cfg))) match {
case Success(_) =>
logger.info("OK")
case Failure(e) =>
logger.info("ERROR")
}
}
}
2.- main_process_fun.scala:
package mod.pack.tools
import mod.pack.tools.another_folder.another_trait
import com.typesafe.config.Config
trait main_process_fun extends another_trait with com.typesafe.scalalogging.LazyLogging {
def pre_defined_f1(conf: Config): Try[Config] = Try {
blablabla
}
def pre_defined_f2(conf: Config)(implicit spark: SparkSession): Try[DataFrame] = Try {
blablabla
logger.info("LOG")
}
def pre_defined_f3(conf: Config): Column = {
blablabla
logger.info("LOG")
}
}
I have an error when trying to compile the project. The error says:
trait main_process inherits conflicting members
lazy value logger in trait LazyLogging of type org.slf4j.Logger and
lazy value logger in trait LazyLogging of type com.typesafe.scalalogging.Logger
(Note: this can be resolved by declaring an override in trait main_process.)
trait main_process extends InitSpark with main_process_fun {
I've already tried to override the logger, but I got this error message:
lazy value logger has incompatible type
I also tried to import main_process_fun instead of extending it from main_process, but without success, I don't know if I did it the wrong way or if the idea itself is bad to solve this issue.
I would greatly appreciate your help to solve this inheritance conflict!
By the way, logger is defined as:
package com.typesafe.scalalogging
trait LazyLogging extends scala.AnyRef {
@scala.transient
protected lazy val logger : com.typesafe.scalalogging.Logger = { /* compiled code */ }
}
The error messages explain that there are two versions of logger
, one in org.slf4j.Logger
and the other in com.typesafe.scalalogging.Logger
. These conflict with each other, so you need to drop one. InitSpark
appears to use the first of these, so use the same type in the second trait as well:
trait main_process_fun extends another_trait with org.slf4j.Logger {