scalajson4s

Scala json4s Deserialization for the abstract trait causes MappingExeption: Unexpected type info RefinedType

My code contains the following entities:

case class Source(uuid: String, `type`: String, parameters: Connector)

sealed trait Connector

case class Snowflake(
    val username: String, 
    val password: String,
    val host: String, 
    val role: Option[String], 
    val warehouse: Option[String], 
    val port: Option[String], 
    val db_name: String,
    val schema: Option[String],
    val activeModal: Option[String],
    val use_ssh: Int, 
    val ssh_ip: Option[String],
    val ssh_port: Option[Int],
    val ssh_user: Option[String]
) extends Connector {
val options = Map(
    ("sfURL" -> s"${host}.snowflakecomputing.com"),
    ("sfUser" -> username),
    ("sfPassword" -> password),
    ("sfRole" -> role),
    ("sfDatabase" -> db_name),
    ("sfSchema" -> schema),
    ("sfWarehouse" -> warehouse))
}

case class MySQL(
    ...
) extends Connector {
    ...
}

case class File(
   ...
) extends Connector {
   ...
}

I'm trying to deserialize json representing different connector types. Json can look like this:

val test = {
   "uuid":"12314sdfds12",    
   "type":"snowflake",   
   "parameters": "{\"use_ssh\": 0,
                   \"schema\": \"YESDATA\",
                   \"activeModal\": \"showflake\", 
                   \"warehouse\": \"COMPUTE_WH\",
                   \"role\": \"DIsdfasROLE\", 
                   \"username\": \"Dixcxf\", 
                   \"password\": \"dfgRvf65&Vyuhj65&\", 
                   \"host\": \"wn789454.east-us-2.azure\",
                   \"db_name\": \"DSPSHARE2\"}" 
}

The problem is that "parameters" field is a string type and json can contain more fields except uuid, type and parameters. I came up with the custom serializer for the Source class:

case object SourceSerializer extends CustomSerializer[Source](format => (
    {
      case JObject(source) =>  {
        
        implicit val formats: Formats = DefaultFormats
        
        def getfield(key: String) = source.filter(field => field match {
            case JField(`key`, JString(_)) => true
            case _ => false
        })

        getfield("uuid") match {
            case List(JField("uuid", JString(uuid))) => {

                val JString(connectorType) = JObject(source) \ "type"
                val JString(connectorParams) = JObject(source) \ "parameters"

                connectorType match {
                    case "snowflake" => Source(uuid, connectorType, parse(connectorParams).extract[Snowflake])
                }
            }
            case Nil => null
        }}
        case JNull => null
    },
    { case op: Source => JString(op.getClass.getSimpleName.replace("$","")) }
))

Finally, I'm trying to use it like this:

parse(test).extract[Source]

It fails and I'm getting the error

[error] org.json4s.MappingException: Unexpected type info RefinedType(ClassSymbol(<refinement>, owner=0, flags=0, info=173 ,None),List(TypeRefType(ThisType(java.lang),java.lang.Object,List()), TypeRefType(ThisType(java.io),java.io.Serializable,List())))
[error]         at org.json4s.reflect.package$.fail(package.scala:56)
[error]         at org.json4s.reflect.ScalaSigReader$.findPrimitive$3(ScalaSigReader.scala:194)
[error]         at org.json4s.reflect.ScalaSigReader$.findArgTypeForField(ScalaSigReader.scala:196)
[error]         at org.json4s.reflect.ScalaSigReader$.readField(ScalaSigReader.scala:77)
[error]         at org.json4s.reflect.Reflector$ClassDescriptorBuilder.$anonfun$fields$3(Reflector.scala:113)
[error]         at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
[error]         at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
[error]         at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
[error]         at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
[error]         at scala.collection.TraversableLike.map(TraversableLike.scala:238)
[error]         at scala.collection.TraversableLike.map$(TraversableLike.scala:231)
[error]         at scala.collection.AbstractTraversable.map(Traversable.scala:108)
[error]         at org.json4s.reflect.Reflector$ClassDescriptorBuilder.fields(Reflector.scala:111)
[error]         at org.json4s.reflect.Reflector$ClassDescriptorBuilder.properties(Reflector.scala:130)
[error]         at org.json4s.reflect.Reflector$ClassDescriptorBuilder.result(Reflector.scala:272)
[error]         at org.json4s.reflect.Reflector$.createDescriptorWithFormats(Reflector.scala:87)
[error]         at org.json4s.reflect.Reflector$.$anonfun$describeWithFormats$1(Reflector.scala:70)
[error]         at org.json4s.reflect.Memo.apply(Memo.scala:12)
[error]         at org.json4s.reflect.Reflector$.describeWithFormats(Reflector.scala:70)
[error]         at org.json4s.Extraction$.$anonfun$extract$10(Extraction.scala:456)
[error]         at org.json4s.Extraction$.$anonfun$customOrElse$1(Extraction.scala:781)
[error]         at scala.PartialFunction.applyOrElse(PartialFunction.scala:127)
[error]         at scala.PartialFunction.applyOrElse$(PartialFunction.scala:126)
[error]         at scala.PartialFunction$$anon$1.applyOrElse(PartialFunction.scala:257)
[error]         at org.json4s.Extraction$.customOrElse(Extraction.scala:781)
[error]         at org.json4s.Extraction$.extract(Extraction.scala:455)
[error]         at org.json4s.Extraction$.extract(Extraction.scala:56)
[error]         at org.json4s.ExtractableJsonAstNode$.extract$extension(ExtractableJsonAstNode.scala:22)
[error]         at org.json4s.jackson.JacksonSerialization.read(Serialization.scala:62)
[error]         at org.json4s.Serialization.read(Serialization.scala:31)
[error]         at org.json4s.Serialization.read$(Serialization.scala:31)
[error]         at org.json4s.jackson.JacksonSerialization.read(Serialization.scala:23)
[error]         at org.divian.entities.ConnectorSerializer$$anonfun$$lessinit$greater$2$$anonfun$apply$3.applyOrElse(Entities.scala:236)
[error]         at org.divian.entities.ConnectorSerializer$$anonfun$$lessinit$greater$2$$anonfun$apply$3.applyOrElse(Entities.scala:223)
[error]         at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:38)
[error]         at org.json4s.CustomSerializer$$anonfun$deserialize$1.applyOrElse(CustomSerializer.scala:11)
[error]         at org.json4s.CustomSerializer$$anonfun$deserialize$1.applyOrElse(CustomSerializer.scala:10)
[error]         at org.json4s.Extraction$.customOrElse(Extraction.scala:781)
[error]         at org.json4s.Extraction$.extract(Extraction.scala:455)
[error]         at org.json4s.Extraction$.extract(Extraction.scala:56)
[error]         at org.json4s.ExtractableJsonAstNode$.extract$extension(ExtractableJsonAstNode.scala:22)
[error]         at org.divian.entities.SourceSerializer$$anonfun$$lessinit$greater$3$$anonfun$apply$5.applyOrElse(Entities.scala:273)
[error]         at org.divian.entities.SourceSerializer$$anonfun$$lessinit$greater$3$$anonfun$apply$5.applyOrElse(Entities.scala:251)
[error]         at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:38)
[error]         at org.json4s.CustomSerializer$$anonfun$deserialize$1.applyOrElse(CustomSerializer.scala:11)
[error]         at org.json4s.CustomSerializer$$anonfun$deserialize$1.applyOrElse(CustomSerializer.scala:10)
[error]         at org.json4s.Extraction$.customOrElse(Extraction.scala:781)
[error]         at org.json4s.Extraction$.extract(Extraction.scala:455)
[error]         at org.json4s.Extraction$.extract(Extraction.scala:56)
[error]         at org.json4s.ExtractableJsonAstNode$.extract$extension(ExtractableJsonAstNode.scala:22)
[error]         at Main$.delayedEndpoint$Main$1(main.scala:48)
[error]         at Main$delayedInit$body.apply(main.scala:21)
[error]         at scala.Function0.apply$mcV$sp(Function0.scala:39)
[error]         at scala.Function0.apply$mcV$sp$(Function0.scala:39)
[error]         at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
[error]         at scala.App.$anonfun$main$1$adapted(App.scala:80)
[error]         at scala.collection.immutable.List.foreach(List.scala:392)
[error]         at scala.App.main(App.scala:80)
[error]         at scala.App.main$(App.scala:78)
[error]         at Main$.main(main.scala:21)
[error]         at Main.main(main.scala)
[error]         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[error]         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[error]         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[error]         at java.lang.reflect.Method.invoke(Method.java:498)
[error] stack trace is suppressed; run last Compile / run for the full output
[error] (Compile / run) org.json4s.MappingException: Unexpected type info RefinedType(ClassSymbol(<refinement>, owner=0, flags=0, info=173 ,None),List(TypeRefType(ThisType(java.lang),java.lang.Object,List()), TypeRefType(ThisType(java.io),java.io.Serializable,List())))
[error] Total time: 16 s, completed Jun 19, 2022 2:14:04 AM

I checked, that:

  1. When Snowflake is not extended from Connector, everything is OK: 
     val ctest = """{"use_ssh": 0, "schema": "YESDATA", "activeModal": "showflake", 
       "warehouse": "COMPUTE_WH", "role": "DIv453SEFROLE", "username": "Disdfg324", 
       "password": "Vgdtj65&stghj65&", "host": "wn354673.east-us-2.azure", "db_name": 
       "DSPSHARE2"}"""
 parse(ctest).extract[Snowflake]
 /* prints correct instance of Snowflake */
  2. It doesnt matter whether I implement it with abstract class, or sealed trait, or trait.
  3. I've tried to use hints instead of the custom serializer for Connector - no success.
  4. Tried to use different versions of json4s (just in case) - no success.

I will really appreciate any suggestions.

Solution

  • It turns out the problem was in the body of Snowflake. Solved the problem by adding .getOrElse("") in the Option[] values of the options map:

    case class Snowflake(
    val username: String, //+
    val password: String, // +
    val host: String, //+
    val role: Option[String],  //+
    val warehouse: Option[String], //+
    val port: Option[String], // String?
    val db_name: String, // +
    val schema: Option[String], //+
    val activeModal: Option[String],
    val use_ssh: Int, // +
    val ssh_ip: Option[String],
    val ssh_port: Option[Int],
    val ssh_user: Option[String]
) extends Connector {
    val options = Map(
        ("sfURL" -> s"${host}.snowflakecomputing.com"),
        ("sfUser" -> username),
        ("sfPassword" -> password),
        ("sfDatabase" -> db_name),
        ("sfRole" -> role.getOrElse("")),
        ("sfSchema" -> schema.getOrElse("")),
        ("sfWarehouse" -> warehouse.getOrElse("")))
}

    But still, it's an interesting behavior. I thought the Option[] constructor val is getting None value if there is no value for it in json. So I don't see a problem with creating a Map with None values.

    For example, I'm able to create something like this:

    val test = Map(
        ("key1" -> "stringValue"),
        ("key2" -> 123),
        ("key3" -> None)
    )

    It will be a Map[String, Any]. Don't understand why it causes the error in my case.

    If someone can explain - it'll be wonderful.