scalaapache-sparkspark-streamingmqttapache-bahir

java.lang.ClassNotFoundException: Failed to find data source: org.apache.bahir.sql.streaming.mqtt.MQTTStreamSinkProvider. Please find packages

I have added

"org.apache.bahir" %% "spark-streaming-mqtt" % "2.4.0"

to my build.sbt, and using

df
  .writeStream
  .format("org.apache.bahir.sql.streaming.mqtt.MQTTStreamSinkProvider")
  .outputMode("complete")
  .option("topic", "mytopic")
  .option("brokerUrl", "tcp://localhost:1883")
  .start()
  .awaitTermination(20000)

in code, but getting

java.lang.ClassNotFoundException: Failed to find data source: org.apache.bahir.sql.streaming.mqtt.MQTTStreamSinkProvider. Please find packages at http://spark.apache.org/third-party-projects.html

    at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:679)
Solution

  • I have been realized that the reason of that behavior is usage of the wrong dependency "org.apache.bahir" %% "spark-streaming-mqtt" % "2.4.0" instead of "org.apache.bahir" %% "spark-sql-streaming-mqtt" % "2.4.0"

    After repairing that I have faced the new issues