node.jsmongodbmongooseexport-to-text

Export large number of documents into text file using mongoose


Im am having about 60,000 records imported into MongoDb using Mongoose schema from different source files. Now I wanted to perform an export into text file of selected records (also applying validations on fields- e.g. there are two fields in the mongoose schema email1 and email2. I have to select between email1 and email2 as a field(Email) in the export file for each records). I referred the plugin mongoose-to-csv. Is there any other plugins or methods available for this export to txt files?

And yes I tried using fs module.

I want the result something like this in a txt document.

"ID","First_Name","Last_Name","Email"
123,ABC, DEF, emailABC@example.com
456,GHI, JKL, emailGDI@exaple.com
.....

my code was

Model.find({"ID":{$exist:true}},function(error,result)
{       
var csv = json2csv({data:results, fields:fields, hasCSVColumnTitle:true});  
fs.writeFile('TestFile.csv',csv,function(err)
{   
});  
});

Can we write these comma separated values into a text file?


Solution

  • I didn't actually tested the code but it should be working fine (maybe with small changes) and should help you to get the idea. This implementation is using streams which will automatically process and write the data. Much more scalable than putting everything in memory. From 2.4.0. version, mongoose supports streams. event-stream is excellent library for manipulating the stream's data.

    var es = require('event-stream');
    var fs = require('fs');
    var wstream = fs.createWriteStream('TestFile.csv');
    
    Model.find({"ID":{$exist:true}).stream()
    .pipe(es.map(function (data, cb) {
          var formated = data;
          formated.email = data.email1 || data.email2;
          var csv = json2csv({data:formated, fields:fields, hasCSVColumnTitle:true});
          cb(null, csv)
        }))
    .pipe(wstream);