Not able to dump JSON response to a FTP server in a CSV file using streamsets

I created a pipeline with HTTP client > Field Pivoter > Field Flattenner > SFTP/FTP/FTPS Client

I am simply trying to fetch data from a HTTP API which returns JSON and dump its response to a FTP server in a CSV file.

When I am trying to preview it "Write to Destination and Executors" I am getting this error

com.streamsets.pipeline.lib.generator.DataGeneratorException: WHOLE_FILE_GENERATOR_ERROR_0 - Whole File Format Error. Reason : java.lang.IllegalArgumentException: Record does not contain the mandatory fields /fileRef,/fileInfo,/fileInfo/size for Whole File Format.

I checked the Docs, there isn't much on what it means and how to resolve this.

In the FTP client destination, which is the final block, I have specified the File Name Expressions as ${record:value('/fileInfo/filename')}.csv

If I dont check the "Write to destination and executor" checkbox while previewing, I can see all the data and its transformation. But when I am trying to dump, it shows that error.

How can I resolve this?

Unfortunately, it is not possible to write CSV data directly from the FTP client destination. One way to do this would be to write data to local disk in one pipeline, and then move the files to the FTP server in a second.

Notes:

• Your first pipeline would use Local FS destination with 'Delimited' data format.
• Your second pipeline would use Directory origin and SFTP/FTP/FTPS Client destination, both set to 'Whole File' data format.