jsoncsvapache-nifihortonworks-dataflow

Convert JSON to CSV in nifi


I want to convert JSON files to CSV in nifi. We can achieve this in Python and other programming languages and have multiple articles on it. I have multiple JSON files and each file has different schema(one specific file will have one schema only). I can see there are templates to convert CSV to JSON and other conversions. But I didn't see any template to convert JSON data to CSV. I have gone through the article https://community.hortonworks.com/articles/64069/converting-a-large-json-file-into-csv.html ,however here we are hard coding the schema. As I have multiple files and each file has different schema, I can't hardcode the schema. Any suggestions please.


Solution

  • Conversion between formats is typically done through ConvertRecord by plugging in the appropriate record reader and record writer, in this case a JSON reader and CSV writer.

    To make use of the record processors you need to defined Avro schemas for your data and put them in a schema registry, NiFi provides a local one.

    There are lots of examples and posts out there about the record stuff, this slide deck shows an example of CSV to JSON, but would be easy to reverse the situation for your scenario:

    https://www.slideshare.net/BryanBende/apache-nifi-record-processing

    This post has some other info: https://bryanbende.com/development/2017/06/20/apache-nifi-records-and-schema-registries