scala apache-spark apache-spark-dataset apache-spark-encoders

Spark Error: Unable to find encoder for type stored in a Dataset

I am using Spark on a Zeppelin notebook, and groupByKey() does not seem to be working.

This code:

df.groupByKey(row => row.getLong(0))
  .mapGroups((key, iterable) => println(key))

Gives me this error (presumably a compilation error, since it shows up in no time while the dataset I am working on is pretty big):

error: Unable to find encoder for type stored in a Dataset.  Primitive types (Int, String, etc) and Product types (case classes) are supported by importing spark.implicits._  Support for serializing other types will be added in future releases.

I tried to add a case class and map all of my rows into it, but still got the same error

import spark.implicits._

case class DFRow(profileId: Long, jobId: String, state: String)

def getDFRow(row: Row):DFRow = {
    return DFRow(row.getLong(row.fieldIndex("item0")),
                 row.getString(row.fieldIndex("item1")), 
                 row.getString(row.fieldIndex("item2")))
}

df.map(DFRow(_))
  .groupByKey(row => row.getLong(0))
  .mapGroups((key, iterable) => println(key))

The schema of my Dataframe is:

root
|-- item0: long (nullable = true)
|-- item1: string (nullable = true)
|-- item2: string (nullable = true)

Solution

You're trying to mapGroups with a function (Long, Iterator[Row]) => Unit and there is no Encoder for Unit (not that it would make sense to have one).

In general parts of the Dataset API which are not focused on the SQL DSL (DataFrame => DataFrame, DataFrame => RelationalGroupedDataset, RelationalGroupedDataset => DataFrame, RelationalGroupedDataset => RelationalGroupedDataset) require either implicit or explicit encoders for the output values.

Since there are no predefined encoders for Row objects, using Dataset[Row] with methods design for statically typed data doesn't make much sense. As a rule of thumb you should always convert to the statically typed variant first:

df.as[(Long, String, String)]