[SOLVED] Read CSV over SSH and convert to JSON

Read CSV over SSH and convert to JSON

This is a duplicate of this question here

Here is the code I'm trying to work with:

let Client = require('ssh2-sftp-client');
let sftp = new Client();
var csv = require("csvtojson");

sftp.connect({
 host: 'HOST',
 port: 'PORT',
 username: 'USERNAME',
 password: 'PASSWORD'
}).then(() => {
 return sftp.get('/home/user/etc/testfile.csv');
  }).then((data) => {
   csv()
   .fromString(data.toString()) // changed this from  .fromStream(data)
   .subscribe(function(jsonObj) { //single json object will be emitted for each csv line
      // parse each json asynchronously
    return new Promise(function(resolve, reject) {
       resolve()
       console.log(jsonObj);
    })
   })
 }).catch((err) => {
 console.log(err, 'catch error');
});

I can read back the CSV data, and can see it going into JSON format on console.log(jsonObj) but the data is unreadable, all '\x00o\x00n\x00'' ..

I'm not sure what to do in the line: // parse each json asynchronously

Could anyone help to figure out how to parse the CSV/JSON after it comes back from the buffer?

Solution

The null bytes \x00 are pointing towards an encoding/decoding issue. The CSV file might be encoded using UTF-16, but Buffer.toString() by default decodes the data using UTF-8. You can change that to data.toString('utf16le') (or data.toString('ucs2')) to force using the correct encoding.