javamavenapache-sparkhadoopgraph

cannot import graphframes dependency in maven project


I have a maven project and i need import graphframe dependency to use spark grapx,this's my pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.example</groupId>
    <artifactId>demoGraphX</artifactId>
    <version>1.0-SNAPSHOT</version>

    <repositories>
        <repository>
            <id>SparkPackagesRepo</id>
            <url>http://dl.bintray.com/spark-packages/maven</url>
        </repository>
    </repositories>

    <dependencies>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-graphx_2.11</artifactId>
            <version>2.2.1</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/graphframes/graphframes -->
        <dependency>
            <groupId>graphframes</groupId>
            <artifactId>graphframes</artifactId>
            <version>0.7.0-spark2.4-s_2.11</version>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_2.11</artifactId>
            <version>2.2.1</version>
        </dependency>
    </dependencies>

    <properties>
        <maven.compiler.source>8</maven.compiler.source>
        <maven.compiler.target>8</maven.compiler.target>
    </properties>

</project>

But when i import graphframe dependency, my IDE return:

Cannot resolve org.apache.parquet:parquet-jackson:1.8.2
Cannot resolve com.twitter:chill_2.11:0.8.0
Cannot resolve org.apache.parquet:parquet-hadoop:1.8.2
Cannot resolve io.dropwizard.metrics:metrics-core:3.1.2
Cannot resolve org.glassfish.hk2:hk2-utils:2.4.0-b34
Cannot resolve com.univocity:univocity-parsers:2.2.1
Cannot resolve org.apache.directory.api:api-asn1-api:1.0.0-M20
Cannot resolve org.glassfish.hk2:osgi-resource-locator:1.0.1
Cannot resolve org.objenesis:objenesis:2.1
Cannot resolve commons-cli:commons-cli:1.2
Cannot resolve org.apache.hadoop:hadoop-yarn-common:2.6.5
Cannot resolve org.mortbay.jetty:jetty-util:6.1.26
Cannot resolve com.google.code.gson:gson:2.2.4
Cannot resolve com.chuusai:shapeless_2.11:2.3.2
etc...

I guest something wrong in my Maven. I use:

Intellij lastest
java 1.8.0_292
Apache Maven 3.6.3

Can you help me solve this problem. Thank you!


Solution

  • The bintray service was shutdown starting from 1st of May. (Press release)

    So Apache spark community has provided new repo to host all spark packages. You can add/replace below code snippet in your code and things should work.

    <repositories>
        <repository>
            <id>bintray</id>
            <name>Bintray Repository</name>
            <url>https://repos.spark-packages.org</url>
        </repository>
    </repositories>