Let's say I have a list of pyspark dataframes: [df1, df2, ...]
, what I want is to union them (so actually do df1.union(df2).union(df3)...
. What's the best practice to achieve that?
you could use the reduce
and pass the union
function along with the list of dataframes.
import pyspark
from functools import reduce
list_of_sdf = [df1, df2, ...]
final_sdf = reduce(pyspark.sql.dataframe.DataFrame.unionByName, list_of_sdf)
the final_sdf
will have the appended data.