eclipsescalaapache-sparkscala-ide

Why eclipse thinks df.as[CaseClass] as an error in Scala Spark program?

I'm trying to convert a dataframe to dataset using the syntax 

case class Schema(...)
val ds = df.as[Schema]

So my code looks like 

case class Rule(rule_on: String, rule_operator: String, rule_value: Int, rule_name: String)
val rules_ds = rules_df
   .select("rule_on", "rule_operator", "rule_value", "rule_name")
   .as[Rule]

But eclipse is highlighting .as[Rule] as error. Screen shots of the same as below.
Error Screen-Shot
How to resolve this issue? I know its not a Scala issue, as it works on command line.

Environment (as in Eclipse):

Solution

  • As suggested by Raphael Roth (in comments) I defined case class outside main method and it works like charm.

    Also other solution (without using case class) is to typecast the dataframe to dataset as below 

    import org.apache.spark.sql._
val ds: Dataset[Row] = df

    The above solution was taken from here