javascriptaxiosbufferlz4

How to create a Buffer in JS to decompress an lz4 file?


I have a file in s3 that is compressed using lz4 (I tried bz2 too) compression and served from an API endpoint as an application/octet-stream. The API fetches the compressed file from S3 and serves it to the frontend where I want to decompress it. The trouble I'm having is with converting the string to a Buffer because I believe the lz4 compressed file contains non-utf8 characters and so I'm unable to create a Uint8Array or a Buffer from it without converting non-utf8 characters to U+FFFD.

Some of the errors I'm seeing:

Error: Invalid magic number
Error: Invalid stream descriptor checksum
RangeError: invalid array length

Does anyone know how to get around this? Is there something simple I'm missing? Thanks in advance!


Solution

  • When I asked this question, I was convinced there was something wrong with the way I was converting things because of the following:

    const sample = 'abcdefghijkl';
    console.log('sample:', sample);
    
    const buffer = Buffer.from(sample);
    console.log('buffer', buffer);
    
    const compressedSample = lz4.encode(sample);
    console.log('compressedSample:', compressedSample);
    const thisWorks = lz4.decode(compressedSample).toString();
    // This works
    console.log('thisWorks', thisWorks);
    
    const compressedSampleString = compressedSample.toString();
    console.log('compressedSample string:\n', compressedSampleString);
    
    const bufferCompressed = Buffer.from(compressedSampleString);
    console.log('bufferCompressed:', bufferCompressed);
    console.log('bufferCompressed string:\n', bufferCompressed.toString());
    
    // This doesn't...
    const decompressedSample = lz4.decode(bufferCompressed);
    

    But it turns out that axios was the issue as decribed here. If I replace axios.get with axios.request and set it to accept binary data, then decoding succeeds:

    public async binaryGet(url: string): Promise<any> {
      try {
        const response = await axios.request({
          method: 'GET',
          url: `${this.config.api_domain}${url}`,
          responseType: 'arraybuffer',
          responseEncoding: 'binary',
        });
        return response.data;
      } catch (error) {
        this.throwAPIError(error);
      }
    }