node.jshttphttpszlib

node https & zlib package: can't parse gzip'ed response from stackoverflow.com


gunzip ing the response doesn't work (I definitely get a gzip response), the headers aren't right, I really don't get why. Here's the demo code below:

const https = require('https')
const zlib = require('zlib')

const options = {
  headers: {
    'Accept-Encoding': 'gzip'
  },
}

https.get('https://stackoverflow.com/', options, resp => {
  let data = "";
  resp.on("data", chunk => {
    data += chunk;
  });

  resp.on("end", () => {
    if (resp.headers['content-encoding'] != 'gzip') {
      console.log('should get gzip')
      process.exit(1)
    }
    console.log('got gzip!')
    zlib.gunzip(data, data_ungzip => {
      console.log(data_ungzip)
    })
  });
})

output:

got gzip!
Error: incorrect header check
    at Zlib.zlibOnError [as onerror] (zlib.js:187:17) {
  errno: -3,
  code: 'Z_DATA_ERROR'
}

Solution

  • The gzipped response is corrupted during the conversion to the string data. The following works:

    const resp_uncompressed = resp.pipe(zlib.createGunzip());
    resp_uncompressed.on("data", ...);
    resp_uncompressed.on("end", ...);