apache-sparkapache-spark-encoders

Rowencoder.apply and rowencoder.encoderfor methods in spark catalyst package

I am trying to create a dataset and I have RDD and a schema of structtype. In spark 3.5.0 and above the Rowencoder.apply method is been removed and have introduced new method called Rowencoder.encoderfor which will return agnosticencoder instead of expressionencoder . So what is the difference between those two encoders and is there any way I can I can cast agnosticencoder to expressionencoder

Dataset dataset=createdataset(RDD,Rowencoder.apply(schema)) But the apply method is not there in spark 3.5.0 and a new method is found called Rowencoder.encoderfor(schema)

Solution

  • You can use row method from Encoders class to get Encoder class object. Dataset dataset= createdataset(RDD,Encoders.row(schema))