scalareflectionscala-macrosscala-reflectavro4s

Implicit resolution fail in reflection with ToolBox


I am trying to generate Avro4s's RecordFormat in reflection based on class path. The following code throws an error.

case class MyCaseClass(a: Int)
println(toolBox.compile {
  toolBox.parse(
    s"""
       |import com.sksamuel.avro4s._
       |import mypackage.MyCaseClass
       |RecordFormat[MyCaseClass]
       |""".stripMargin
  )
}())

could not find implicit value for evidence parameter of type com.sksamuel.avro4s.Decoder[mypackage.MyCaseClass]

RecordFormat is like

object RecordFormat {

  def apply[T <: Product : Encoder : Decoder : SchemaFor]: RecordFormat[T] = apply(AvroSchema[T])

  def apply[T <: Product : Encoder : Decoder](schema: Schema): RecordFormat[T] = new RecordFormat[T] {
    private val fromRecord = FromRecord[T](schema)
    private val toRecord = ToRecord[T](schema)
    override def from(record: GenericRecord): T = fromRecord.from(record)
    override def to(t: T): Record = toRecord.to(t)
  }
}

Ref: https://github.com/sksamuel/avro4s/blob/release/2.0.x/avro4s-core/src/main/scala/com/sksamuel/avro4s/RecordFormat.scala

I can see, it can resolve Encoder[MyCaseClass] and SchemaFor[MyCaseClass] but fails for Decoder[MyCaseClass].

The same code can resolve RecordFormat[MyCaseClass] without reflection.

I can see that Decoder is implemented with macro similar to Encoder.

implicit def applyMacro[T <: Product]: Decoder[T] = macro applyMacroImpl[T]

Why reflection cannot resolve the implicit evidence?


Solution

  • avro4s 4.x uses Magnolia but avro4s 2.x uses raw implicit macros + Shapeless.

    Normally there shouldn't be significant problems with materializing a type class at runtime using reflective toolbox even if the type class is defined with macros.

    The issue is now that the macro defining com.sksamuel.avro4s.Decoder has a bug. The line Decoder.scala#L404

    c.Expr[Decoder[T]](
      q"""
      new _root_.com.sksamuel.avro4s.Decoder[$tpe] {
        private[this] val decoders = Array(..$decoders)
    
        override def decode(value: Any, schema: _root_.org.apache.avro.Schema): $tpe = {
          val fullName = $fullName
          value match {
            case record: _root_.org.apache.avro.generic.GenericRecord => $companion.apply(..$fields)
            case _ => sys.error("This decoder decodes GenericRecord => " + fullName + " but has been invoked with " + value)
          }
        }
      }
      """
    )
    

    refers to sys.error instead of hygienic _root_.scala.sys.error.

    If you fix this line, Decoder[MyCaseClass] and RecordFormat[MyCaseClass] will work inside toolbox

    println(toolBox.compile {
      toolBox.parse(
        s"""
           |import com.sksamuel.avro4s._
           |import mypackage.MyCaseClass
           |RecordFormat[MyCaseClass]
           |""".stripMargin
      )
    }()) //com.sksamuel.avro4s.RecordFormat$$anon$1@25109d84
    

    So a fast fix is to remove the line

    libraryDependencies += "com.sksamuel.avro4s" %% "avro4s-core" % "2........."
    

    in build.sbt, add

    libraryDependencies += "com.chuusai" %% "shapeless" % "2.3.3"
    libraryDependencies += "org.apache.avro" % "avro" % "1.8.2"
    

    (otherwise you'll have NoClassDefFoundError) and put the following patched jars into lib

    https://github.com/DmytroMitin/avro4s-2.0.5-2.11-patched

    avro4s-core_2.11-2.0.5-SNAPSHOT.jar
    avro4s-macros_2.11-2.0.5-SNAPSHOT.jar
    

    You can always debug implicit-based or macro-based code generated with toolbox if you create it like

    val toolBox = runtimeMirror.mkToolBox(
      frontEnd = new FrontEnd {
        override def display(info: Info): Unit = println(info)
        override def interactive(): Unit = ???
      },
      options = "-Xlog-implicits" // or "-Xlog-implicits -Ymacro-debug-lite"
    )
    

    If you do

    println(reify{
      Decoder[MyCaseClass]
    }.tree)
    

    it prints

    Decoder.apply[MyCaseClass](Decoder.applyMacro)
    

    so implicit Decoder[MyCaseClass] is resolved as Decoder.applyMacro[MyCaseClass].

    With the original unpatched jars

    toolBox.compile {
      toolBox.parse(
        s"""
           |import com.sksamuel.avro4s._
           |import mypackage.MyCaseClass
           |Decoder.applyMacro[MyCaseClass]
           |""".stripMargin
      )
    }()
    

    threw

    scala.tools.reflect.ToolBoxError: reflective compilation has failed:
    object error is not a member of package sys