scalaapache-sparkapache-spark-sqlrlike

How to extract column value to compare with rlike in spark dataframe


I have the below code in spark 2.3 .It shows error since rlike can only evaluate a string but not a column field, So I want to extract the value in corresponding col(unique_handle_regex) column as string

val df3 = df1.join(df2).select("*").where(col(unique_handle) rlike col(unique_handle_regex))

Solution

  • Try below code.

    val whereExpr = Seq("unique_handle","unique_handle_regex").mkString(" rlike ")
    val df3 = df1.join(df2).select("*").where(whereExpr)