apache-sparkapache-spark-sqldependenciesapache-calcite

typesafe joins in spark for 3.x


In old versions of Spark 3.x

import org.apache.calcite.sql.JoinType

Calcite was exposed. That meant a JOIN could not only be executed in a strongly typed fashion for dataframes (notice I do not mean datasets) by feeding: JoinType.LEFT.name.

Calcite is no longer exposed for 3.x.

Therefore, currently one would need to fall back to stringly typed join types (which is a bad thing I think). Is there a similar possibility for another class I am overlooking (which actually is exposed)? I am a bit reluctant to manually add calcite in my user code as this might result in conflicting classes (but it would be a viable option).


Solution

  • import org.apache.hadoop.hive.ql.parse.JoinType 
    

    seems to do the trick