dataframescalaapache-sparkapache-spark-sqldayofweek

How to extract week day as a number from a Spark dataframe with the Scala API

I have a date column which is string in dataframe in the 2017-01-01 12:15:43 timestamp format.

I want to get weekday number (1 to 7) from that column using dataframe and Scala (not Spark SQL).

Like below

df.select(weekday(col("colname")))

I found one in Python and SQL, but not in Scala. Can any body help me on this

in sqlcontext

sqlContext.sql("select date_format(to_date('2017-01-01'),'W') as week")
Solution

  • This works the same way in Scala:

    scala> spark.version
res1: String = 2.3.0

scala> spark.sql("select date_format(to_date('2017-01-01'),'W') as week").show
// +----+
// |week|
// +----+
// |   1|
// +----+

    or

    scala> import org.apache.spark.sql.functions._
import org.apache.spark.sql.functions._

scala> val df = Seq("2017-01-01").toDF("date")
df: org.apache.spark.sql.DataFrame = [date: string]

scala> df.select(date_format(to_date(col("date")), "W")).show
// +-------------------------------+
// |date_format(to_date(`date`), W)|
// +-------------------------------+
// |                              1|
// +-------------------------------+