apache-spark-sqlhiveinsertspark3

Will Insert Into Command preserve order in Spark3

Would like to know if the columns will be inserted into the Hive table in the correct position when I use insert into the target table select * from the temp table created in spark.

I am trying to insert into a partitioned table in hive from spark. I do a select on the df and then create a temp table and then insert into target from the temp table. Would like to know if it will preserve the Order.

df.select("col1","col2","col3","YearMonth").createOrReplaceTempView("Table1")
spark.sql("insert into TargetTable partition(yyyymm) select * from Table1")
Solution

  • Yes, order is maintained. But in addition, If your TargetTable has same no of columns and corresponding datatypes from select * query, then there wont be a problem