node.jsstreamfast-csv

How to identify different input schemas with FastCSV?


The following code works fine with the "Winner" type. The tech is typescript with node streams.

Sometimes someone uploads a winner-2 type though. I'd like to look at the header and change format type based on the header.

I could possibly

I'm planning on rewriting the headers as there are inconsistencies.

How to solve the problem of normalising these different CSV inputs into one idealised structure? rxjs?

import {parse, RowTransformCallback} from "@fast-csv/parse";

 stream
     .pipe(parse({headers: true}))
     .pipe(format<Winner, Winner>({headers: true}))
     .transform((row, next): void => {
           this.processRow(row, next)
  })
     .pipe(process.stdout)
     .on('error', reject)
            .on('end',
                (rowCount: number) => console.log(`Parsed ${rowCount} rows`));
    });

Solution

  • I rewrote the headers in a map function to take schema a and schema b and turn them into the target schema

     stream
        .pipe(parse({
          headers: headers => headers.map(h => {
            if (h === 'Email') {
               return 'email'
            }
            if (h === 'Firstname') {
               return 'firstName'                  
            }
            return h
         }),
        }))