apache-sparkpysparkspark-graphx

Is Graph available on pyspark for Spark 3.0+


I was wondering if GraphX API is available in PySpark for Spark 3.0+? I'm not finding any of that sort in official documentation. All the examples are developed with Scala. And Where can I get more updates about it.

Thanks, Darshan


Solution

  • According to the documentation available at http://ampcamp.berkeley.edu/big-data-mini-course/graph-analytics-with-graphx.html:

    "The GraphX API is currently only available in Scala but we plan to provide Java and Python bindings in the future."

    However, you should look at GraphFrames (https://github.com/graphframes/graphframes), which wraps GraphX algorithms under the DataFrames API and it provides Python interface.

    Here is a quick example from https://graphframes.github.io/graphframes/docs/_site/quick-start.html, with slight modification so that it works.

    First, start pyspark with the graphframes pkg loaded.

    pyspark --packages graphframes:graphframes:0.1.0-spark1.6

    python code:

    from graphframes import *
    
    # Create a Vertex DataFrame with unique ID column "id"
    v = sqlContext.createDataFrame([
      ("a", "Alice", 34),
      ("b", "Bob", 36),
      ("c", "Charlie", 30),
    ], ["id", "name", "age"])
    
    # Create an Edge DataFrame with "src" and "dst" columns
    e = sqlContext.createDataFrame([
      ("a", "b", "friend"),
      ("b", "c", "follow"),
      ("c", "b", "follow"),
    ], ["src", "dst", "relationship"])
    # Create a GraphFrame
    g = GraphFrame(v, e)
    
    # Query: Get in-degree of each vertex.
    g.inDegrees.show()
    
    # Query: Count the number of "follow" connections in the graph.
    g.edges.filter("relationship = 'follow'").count()
    
    # Run PageRank algorithm, and show results.
    results = g.pageRank(resetProbability=0.01, maxIter=20)
    results.vertices.select("id", "pagerank").show()