I have a scenario where I am reading from my hive table and creating a spark dataframe. I want to generate an xml string from the output of dataframe and save it in a new dataframe (as xml string) , rather than writing it to a file in HDFS to create an xml. Please tell me if this can be done using databricks spark-xml.
You cannot do this with the spark-xml lib, but you can reuse the write out part from it to create your own solution for an XmlRdd: https://github.com/databricks/spark-xml/blob/master/src/main/scala/com/databricks/spark/xml/util/XmlFile.scala
You can find in line 80 exactly this.