node.jsgoogle-bigquerygoogle-cloud-storageparquetparquetjs

Parquet write to gcs is not queryable by bigquery in nodejs

i'm using parquetjs to create parquet files and push to google cloud storage.

Problem is that bigquery cannot read the data from file but when i use parquet-tools everything looks healthy.

Solution

  • just pass useDataPageV2: false as option to parquet.ParquetWriter.openFile(...)

    Like this:

    const writer = await parquet.ParquetWriter.openFile(schema, filename, { useDataPageV2: false })