I have a case class generated from a .proto file through scalapb, which has a few fields with bcl.DateTime type. The case class definition is as follows:
@SerialVersionUID(0L)
final case class EditorialAdEntity(
customerid: _root_.scala.Int = 0,
accountid: _root_.scala.Int = 0,
orderId: _root_.scala.Long = 0L,
entityId: _root_.scala.Long = 0L,
dataFeedId: _root_.scala.Long = 0L,
editorialStatusModifiedDTim: _root_.scala.Option[bcl.bcl.DateTime] = _root_.scala.None,
modifiedDTim: _root_.scala.Option[bcl.bcl.DateTime] = _root_.scala.None,
adTitle: _root_.scala.Predef.String = "",
adDescription: _root_.scala.Predef.String = "",
adDescription2: _root_.scala.Predef.String = "",
displayURL: _root_.scala.Predef.String = "",
businessName: _root_.scala.Predef.String = "",
...
I am able to create an instance of this case class and view the contents as follows:
val currentDt: DateTime = DateTime.of(value = Some(DateTimeUtils.getCurrentMillis), kind = Some(DateTimeKind.UTC), scale = Some(TimeSpanScale.MILLISECONDS))
val entity: EditorialAdEntity = EditorialAdEntity(customerid = customerId, accountid = accountId, adTitle = "test",
orderId = orderId, serviceLevelId = 5, campaignType = campaignType,
createdDtim = Some(currentDt), modifiedDTim = Some(currentDt),
editorialStatusModifiedDTim = Some(currentDt) )
Logger.logInfo(entity.toProtoString)
However, when I create a Spark dataframe on top of this, as follows:
val data = spark.sqlContext.createDataFrame(List(entity))
data.show()
I get the following error:
Exception in thread "main" scala.ScalaReflectionException: <none> is not a term
at scala.reflect.api.Symbols$SymbolApi$class.asTerm(Symbols.scala:199)
at scala.reflect.internal.Symbols$SymbolContextApiImpl.asTerm(Symbols.scala:84)
at org.apache.spark.sql.catalyst.ScalaReflection$class.constructParams(ScalaReflection.scala:985)
at org.apache.spark.sql.catalyst.ScalaReflection$.constructParams(ScalaReflection.scala:46)
at org.apache.spark.sql.catalyst.ScalaReflection$class.getConstructorParameters(ScalaReflection.scala:965)
at org.apache.spark.sql.catalyst.ScalaReflection$.getConstructorParameters(ScalaReflection.scala:46)
at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$schemaFor$1.apply(ScalaReflection.scala:782)
at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$schemaFor$1.apply(ScalaReflection.scala:724)
at scala.reflect.internal.tpe.TypeConstraints$UndoLog.undo(TypeConstraints.scala:56)
at org.apache.spark.sql.catalyst.ScalaReflection$class.cleanUpReflectionObjects(ScalaReflection.scala:906)
at org.apache.spark.sql.catalyst.ScalaReflection$.cleanUpReflectionObjects(ScalaReflection.scala:46)
at org.apache.spark.sql.catalyst.ScalaReflection$.schemaFor(ScalaReflection.scala:723)
at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$schemaFor$1.apply(ScalaReflection.scala:737)
at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$schemaFor$1.apply(ScalaReflection.scala:724)
at scala.reflect.internal.tpe.TypeConstraints$UndoLog.undo(TypeConstraints.scala:56)
at org.apache.spark.sql.catalyst.ScalaReflection$class.cleanUpReflectionObjects(ScalaReflection.scala:906)
at org.apache.spark.sql.catalyst.ScalaReflection$.cleanUpReflectionObjects(ScalaReflection.scala:46)
at org.apache.spark.sql.catalyst.ScalaReflection$.schemaFor(ScalaReflection.scala:723)
at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$schemaFor$1$$anonfun$apply$8.apply(ScalaReflection.scala:785)
at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$schemaFor$1$$anonfun$apply$8.apply(ScalaReflection.scala:784)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.immutable.List.foreach(List.scala:381)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.immutable.List.map(List.scala:285)
at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$schemaFor$1.apply(ScalaReflection.scala:784)
at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$schemaFor$1.apply(ScalaReflection.scala:724)
at scala.reflect.internal.tpe.TypeConstraints$UndoLog.undo(TypeConstraints.scala:56)
at org.apache.spark.sql.catalyst.ScalaReflection$class.cleanUpReflectionObjects(ScalaReflection.scala:906)
at org.apache.spark.sql.catalyst.ScalaReflection$.cleanUpReflectionObjects(ScalaReflection.scala:46)
at org.apache.spark.sql.catalyst.ScalaReflection$.schemaFor(ScalaReflection.scala:723)
at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$schemaFor$1.apply(ScalaReflection.scala:737)
at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$schemaFor$1.apply(ScalaReflection.scala:724)
at scala.reflect.internal.tpe.TypeConstraints$UndoLog.undo(TypeConstraints.scala:56)
at org.apache.spark.sql.catalyst.ScalaReflection$class.cleanUpReflectionObjects(ScalaReflection.scala:906)
at org.apache.spark.sql.catalyst.ScalaReflection$.cleanUpReflectionObjects(ScalaReflection.scala:46)
at org.apache.spark.sql.catalyst.ScalaReflection$.schemaFor(ScalaReflection.scala:723)
at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$schemaFor$1$$anonfun$apply$8.apply(ScalaReflection.scala:785)
at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$schemaFor$1$$anonfun$apply$8.apply(ScalaReflection.scala:784)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.immutable.List.foreach(List.scala:381)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.immutable.List.map(List.scala:285)
at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$schemaFor$1.apply(ScalaReflection.scala:784)
at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$schemaFor$1.apply(ScalaReflection.scala:724)
at scala.reflect.internal.tpe.TypeConstraints$UndoLog.undo(TypeConstraints.scala:56)
at org.apache.spark.sql.catalyst.ScalaReflection$class.cleanUpReflectionObjects(ScalaReflection.scala:906)
at org.apache.spark.sql.catalyst.ScalaReflection$.cleanUpReflectionObjects(ScalaReflection.scala:46)
at org.apache.spark.sql.catalyst.ScalaReflection$.schemaFor(ScalaReflection.scala:723)
at org.apache.spark.sql.catalyst.ScalaReflection$.schemaFor(ScalaReflection.scala:720)
at org.apache.spark.sql.SparkSession.createDataFrame(SparkSession.scala:313)
at org.apache.spark.sql.SQLContext.createDataFrame(SQLContext.scala:285)
at Scripts.SampleScripts.Protobuf.demo.EnforcementProtoTester$.main(EnforcementProtoTester.scala:43)
at Scripts.SampleScripts.Protobuf.demo.EnforcementProtoTester.main(EnforcementProtoTester.scala)
If I remove the DateTime fields from the proto class it seems to work fine. Any pointers on how to create dataframes on top of proto classes withbcl.DateTime fields?
In order to use ScalaPB generated class with Spark, you need to add a library dependency on sparksql-scalapb, and use ProtoSQL.createDataFrame() instead of spark.sqlContext.createDataFrame. The process is described here: https://scalapb.github.io/sparksql.html#using-sparksql-scalapb