scalaapache-sparkapache-spark-datasetapache-spark-encoders

Encode an ADT / sealed trait hierarchy into Spark DataSet column

If I want to store an Algebraic Data Type (ADT) (ie a Scala sealed trait hierarchy) within a Spark DataSet column, what is the best encoding strategy?

For example, if I have an ADT where the leaf types store different kinds of data:

sealed trait Occupation
case object SoftwareEngineer extends Occupation
case class Wizard(level: Int) extends Occupation
case class Other(description: String) extends Occupation

Whats the best way to construct a:

org.apache.spark.sql.DataSet[Occupation]
Solution

  • TL;DR There is no good solution right now, and given Spark SQL / Dataset implementation, it is unlikely there will be one in the foreseeable future.

    You can use generic kryo or java encoder

    val occupation: Seq[Occupation] = Seq(SoftwareEngineer, Wizard(1), Other("foo"))
spark.createDataset(occupation)(org.apache.spark.sql.Encoders.kryo[Occupation])

    but is hardly useful in practice.

    UDT API provides another possible approach as for now (Spark 1.6, 2.0, 2.1-SNAPSHOT) it is private and requires quite a lot boilerplate code (you can check o.a.s.ml.linalg.VectorUDT to see example implementation).