I have a python dict as shown below
cam = {"emp_id":1234, "emp_acct": [6784, 8901], "start_date":"2002-05-06"}
I want to create a new Dataframe by add this as a new column to an existing dataframe and I tried
df = input_df.withColumn("cam_details", lit(cam))
This throws as error as LIT takes only a string..
As you haven't shared the expected output schema - here's what I think you want:
from pyspark.sql import SparkSession
import pyspark.sql.functions as F
spark = SparkSession.builder.getOrCreate()
input_df = spark.createDataFrame([(1,)], "id int")
cam = {"emp_id":1234, "emp_acct": [6784, 8901], "start_date":"2002-05-06"}
cam_schema = "struct<emp_id:string, emp_acct:array<int>, start_date:date>"
df = df.withColumn("cam_details", F.from_json(F.lit(str(cam)), cam_schema))
df.printSchema()
df.show(truncate=False)
Output:
root
|-- id: integer (nullable = true)
|-- cam_details: struct (nullable = true)
| |-- emp_id: string (nullable = true)
| |-- emp_acct: array (nullable = true)
| | |-- element: integer (containsNull = true)
| |-- start_date: date (nullable = true)
+---+--------------------------------+
|id |cam_details |
+---+--------------------------------+
|1 |{1234, [6784, 8901], 2002-05-06}|
+---+--------------------------------+