I have the below code in spark 2.3 .It shows error since rlike can only evaluate a string but not a column field, So I want to extract the value in corresponding col(unique_handle_regex) column as string
val df3 = df1.join(df2).select("*").where(col(unique_handle) rlike col(unique_handle_regex))
Try below code.
val whereExpr = Seq("unique_handle","unique_handle_regex").mkString(" rlike ")
val df3 = df1.join(df2).select("*").where(whereExpr)