I'm new to Apache Spark and am attempting to use Java Spring Boot to read a CSV file from Apache Spark. however, I kept getting the below exception despite trying several different approaches.
Is there a need for additional configuration?
SparkSession spark = SparkSession
.builder()
.appName("Java Spark SQL Example")
.getOrCreate();
StructType schema = new StructType()
.add("department", "string")
.add("designation", "string")
.add("ctc", "long")
.add("state", "string");
Dataset<Row> df = spark.read()
.option("mode", "DROPMALFORMED")
.schema(schema)
.csv("C://exampl/bands-csv.csv");
java.io.FileNotFoundException: java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset. -see https://wiki.apache.org/hadoop/WindowsProblems
at org.apache.hadoop.util.Shell.fileNotFoundException(Shell.java:547) ~[hadoop-client-api-3.3.2.jar:na]
at org.apache.hadoop.util.Shell.getHadoopHomeDir(Shell.java:568) ~[hadoop-client-api-3.3.2.jar:na]
at org.apache.hadoop.util.Shell.getQualifiedBin(Shell.java:591) ~[hadoop-client-api-3.3.2.jar:na]
at org.apache.hadoop.util.Shell.<clinit>(Shell.java:688) ~[hadoop-client-api-3.3.2.jar:na]
at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:79) ~[hadoop-client-api-3.3.2.jar:na]
Caused by: java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset.
at org.apache.hadoop.util.Shell.checkHadoopHomeInner(Shell.java:467) ~[hadoop-client-api-3.3.2.jar:na]
at org.apache.hadoop.util.Shell.checkHadoopHome(Shell.java:438) ~[hadoop-client-api-3.3.2.jar:na]
at org.apache.hadoop.util.Shell.<clinit>(Shell.java:515) ~[hadoop-client-api-3.3.2.jar:na]
... 21 common frames omitted
2022-10-10 09:51:44.030 WARN 20884 --- [ restartedMain] org.apache.hadoop.util.NativeCodeLoader : Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2022-10-10 09:51:44.168 ERROR 20884 --- [ restartedMain] org.apache.spark.SparkContext : Error initializing SparkContext.
org.apache.spark.SparkException: A master URL must be set in your configuration
at org.apache.spark.SparkContext.<init>(SparkContext.scala:396) ~[spark-core_2.13-3.3.0.jar:3.3.0]
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2704) ~[spark-core_2.13-3.3.0.jar:3.3.0]
at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:953) ~[spark-sql_2.13-3.3.0.jar:3.3.0]
at scala.Option.getOrElse(Option.scala:201) ~[scala-library-2.13.8.jar:na]
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:947) ~[spark-sql_2.13-3.3.0.jar:3.3.0]
at com.spark.sql.SparkPocApplication.main(SparkPocApplication.java:42) ~[classes/:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Method.java:568) ~[na:na]
at org.springframework.boot.devtools.restart.RestartLauncher.run(RestartLauncher.java:49) ~[spring-boot-devtools-2.7.4.jar:2.7.4]
2022-10-10 09:51:44.229 INFO 20884 --- [ restartedMain] org.apache.spark.SparkContext : Successfully stopped SparkContext
Exception in thread "restartedMain" java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Dependencies added apche-spark core and apache-spark-sql
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.13</artifactId>
<version>3.3.0</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-sql -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.13</artifactId>
<version>3.3.0</version>
<scope>provided</scope>
</dependency>
Issue has got resolved, it was missing configuration of HADOOP_HOME directory and winutils WINUTILS, and hadoop.dll HADOOP.DLL
Solution and configuration can find here : solution