scalaapache-sparkapache-kafkasbtspark-structured-streaming

spark streaming and kafka integration dependency problem

am trying to write a script that reads data from kafka with spark streaming but when i run "sbt compile" i get this error:

sbt.librarymanagement.ResolveException: Error downloading org.apache.spark:spark-sql-kafka-0-10_3:3.5.0

any help would be appreciated

if this has anything to do with version compatibility what are the possibe solutions for this problem.

Solution

  • _3 mentioned in the error isn't a valid version for that package

    This is correct for sbt

    libraryDependencies += "org.apache.spark" %% "spark-sql-kafka-0-10" % "3.5.0"

    But make sure you have Scala 2.13 installed, not Scala 3, as Spark doesn't support that