javaapache-sparkdictionaryapache-spark-encoders

Is there an Encoder for Map type in Java Spark?

I am trying to create a custom Aggregator function producing a Map as the result, however it requires an Encoders. As referenced in https://spark.apache.org/docs/2.1.0/api/java/org/apache/spark/sql/Encoders.html, there isn't one for now.

Does anyone know a workaround for this? Thank you in advance!

Solution

  • This is not exactly an answer, but if someone came here with the same question but in Scala instead of Java, they can use the ExpressionEncoder:

    import org.apache.spark.sql.catalyst.encoders.ExpressionEncoder

// ...

def outputEncoder: Encoder[Map[String, Int]] = ExpressionEncoder()

    For the original question in Java, this link about how to write a custom encoder might help.