javascriptnode.jscsvpapaparsenode-csv-parse

Saving data from a CSV in a global variable (fast-csv/papaparse)


I am new to js and trying to parse a CSV in the backend using node.js.

I have an array of states in which I want to store the data of a column of CSV. This is a very simple code that i wrote using fast-csv to do so. But whenever I run the code, I get an empty array, [] . I tried doing the same using papaparse and got the same results.

const csv = require('fast-csv')
const file = fs.createReadStream('main.csv');
var states = []
file
    .pipe(csv.parse({ headers: false }))
    .on('data', row => states.push(row[2]))
console.log(states)

But whenever I console log it in then .on('end') block the values are logged.

const csv = require('fast-csv')
const file = fs.createReadStream('main.csv');
var states = []
file
    .pipe(csv.parse({ headers: false }))
    .on('data', row => states.push(row[2]))
    .on('end', () => console.log(states) // This works
console.log(states) // This doesn't

I think this is due to the parser working asynchronously. I have tried to resolve promises and used async/await methods but I cant use the parsed content in the global scope.

Would love some help on this one.


Solution

  • You are right, parser works asynchronously. So, it starts parsing and uses callback on events ('data', 'end' in your case). But code after your parser will be executed emediately after parser started to work. So all your actions with parsed data should be done in 'end' event callback.

    // function to start parser
    const startParsing = (res) => {
      const csv = require('fast-csv')
      const file = fs.createReadStream('main.csv');
      // you may use const cos it's type won't be changed
      const states = []
      file
        .pipe(csv.parse({ headers: false }))
        .on('data', row => states.push(row[2]))
        // execute function after parsing.
        .on('end', () => outputData(states, res));
    };
    
    const outputData = (states, res) => {
      // your next actions here
      console.log(states);
      // for example, res.send(states) or anything to complete server response if it's accessible
      res.send(states);  
    };
    
    // then, in any place you need use this startParsing function, for example
    server.get('/parser', (req, res) => {
      startParsing(res);
    });