I have been experimenting with multiple spark sessions using the same spark context. Once the spark-shell is launched, I can create a new spark session using the following.
val spark2 = spark.newSession();
spark2: org.apache.spark.sql.SparkSession = org.apache.spark.sql.SparkSession@43e869ea
scala> spark
res0: org.apache.spark.sql.SparkSession = org.apache.spark.sql.SparkSession@1390db9e
scala> spark2
res1: org.apache.spark.sql.SparkSession = org.apache.spark.sql.SparkSession@43e869ea
However, even though the session is created and available, it is still not an active session. Each spark session has its own temporary views. If I am creating a data frame, it is already associated with the default spark session.
scala> val s = Seq(1,2,3).toDF("num")
s: org.apache.spark.sql.DataFrame = [num: int]
scala> val s2 = Seq(4,5,6).toDF("num")
s2: org.apache.spark.sql.DataFrame = [num: int]
scala> s
res2: org.apache.spark.sql.DataFrame = [num: int]
scala> s.sparkSession
res3: org.apache.spark.sql.SparkSession = org.apache.spark.sql.SparkSession@1390db9e
Is there a way to switch the active spark session, so that the next operations are done using that?
You're probably looking for the setActiveSession(session: SparkSession): Unit method defined in companion object.
scala> val spark2 = spark.newSession();
spark2: org.apache.spark.sql.SparkSession = org.apache.spark.sql.SparkSession@43e869ea
scala> import org.apache.spark.sql.SparkSession
scala> SparkSession.setActiveSession(spark2)
scala> SparkSession.active
res1: org.apache.spark.sql.SparkSession = org.apache.spark.sql.SparkSession@43e869ea