I am trying to install PySpark package Graphframes using spark-shell :
pyspark --packages graphframes:graphframes:0.8.1-spark3.0-s_2.12
However, there is any error like this in the terminal:
root@hpcc:~# pyspark --packages graphframes:graphframes:0.8.1-spark3.0-s_2.12
Python 3.6.9 (default, Jan 26 2021, 15:33:00)
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/root/spark-3.0.2-bin-hadoop3.2/jars/spark-unsafe_2.12-3.0.2.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Ivy Default Cache set to: /root/.ivy2/cache
The jars for the packages stored in: /root/.ivy2/jars
:: loading settings :: url = jar:file:/root/spark-3.0.2-bin-hadoop3.2/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
graphframes#graphframes added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent-bb0fc7e9-5af7-4189-98e4-7ac76a8d97a9;1.0
confs: [default]
:: resolution report :: resolve 2691ms :: artifacts dl 1ms
:: modules in use:
---------------------------------------------------------------------
| | modules || artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
| default | 1 | 0 | 0 | 0 || 0 | 0 |
---------------------------------------------------------------------
:: problems summary ::
:::: WARNINGS
module not found: graphframes#graphframes;0.8.1-spark3.0-s_2.12
==== local-m2-cache: tried
file:/root/.m2/repository/graphframes/graphframes/0.8.1-spark3.0-s_2.12/graphframes-0.8.1-spark3.0-s_2.12.pom
-- artifact graphframes#graphframes;0.8.1-spark3.0-s_2.12!graphframes.jar:
file:/root/.m2/repository/graphframes/graphframes/0.8.1-spark3.0-s_2.12/graphframes-0.8.1-spark3.0-s_2.12.jar
==== local-ivy-cache: tried
/root/.ivy2/local/graphframes/graphframes/0.8.1-spark3.0-s_2.12/ivys/ivy.xml
-- artifact graphframes#graphframes;0.8.1-spark3.0-s_2.12!graphframes.jar:
/root/.ivy2/local/graphframes/graphframes/0.8.1-spark3.0-s_2.12/jars/graphframes.jar
==== central: tried
https://repo1.maven.org/maven2/graphframes/graphframes/0.8.1-spark3.0-s_2.12/graphframes-0.8.1-spark3.0-s_2.12.pom
-- artifact graphframes#graphframes;0.8.1-spark3.0-s_2.12!graphframes.jar:
https://repo1.maven.org/maven2/graphframes/graphframes/0.8.1-spark3.0-s_2.12/graphframes-0.8.1-spark3.0-s_2.12.jar
==== spark-packages: tried
https://dl.bintray.com/spark-packages/maven/graphframes/graphframes/0.8.1-spark3.0-s_2.12/graphframes-0.8.1-spark3.0-s_2.12.pom
-- artifact graphframes#graphframes;0.8.1-spark3.0-s_2.12!graphframes.jar:
https://dl.bintray.com/spark-packages/maven/graphframes/graphframes/0.8.1-spark3.0-s_2.12/graphframes-0.8.1-spark3.0-s_2.12.jar
::::::::::::::::::::::::::::::::::::::::::::::
:: UNRESOLVED DEPENDENCIES ::
::::::::::::::::::::::::::::::::::::::::::::::
:: graphframes#graphframes;0.8.1-spark3.0-s_2.12: not found
::::::::::::::::::::::::::::::::::::::::::::::
:: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
Exception in thread "main" java.lang.RuntimeException: [unresolved dependency: graphframes#graphframes;0.8.1-spark3.0-s_2.12: not found]
at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1389)
at org.apache.spark.deploy.DependencyUtils$.resolveMavenDependencies(DependencyUtils.scala:54)
at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:308)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:871)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Traceback (most recent call last):
File "/root/spark-3.0.2-bin-hadoop3.2/python/pyspark/shell.py", line 38, in <module>
SparkContext._ensure_initialized()
File "/root/spark-3.0.2-bin-hadoop3.2/python/pyspark/context.py", line 327, in _ensure_initialized
SparkContext._gateway = gateway or launch_gateway(conf)
File "/root/spark-3.0.2-bin-hadoop3.2/python/pyspark/java_gateway.py", line 105, in launch_gateway
raise Exception("Java gateway process exited before sending its port number")
Exception: Java gateway process exited before sending its port number
>>> quit()
root@hpcc:~#
I am using Ubuntu OS 18.04.5 LTS
JDK version is 11.0.11
Scala version is 2.12.13
Spark-shel version is 3.0.2
I am wondering what is the problem ? and How can I overcome the problem ?
The jar has to be downloaded from repos.spark-packages.org. Unfortunately this repo is not checked by pyspark when using the --packages parameter. If your machine has a running Maven installation available, the easiest way to solve the problem is to manually download the jar to your local Maven repository:
mvn org.apache.maven.plugins:maven-dependency-plugin:2.1:get
-Dartifact=graphframes:graphframes:0.8.1-spark3.0-s_2.12
-DrepoUrl=https://repos.spark-packages.org
This command will download the jar (and all required dependencies, if any) to your local Maven repository at /root/.m2/repository. From this location pyspark can pick up the jar.