I have a file in s3 that is compressed using lz4 (I tried bz2 too) compression and served from an API endpoint as an application/octet-stream. The API fetches the compressed file from S3 and serves it to the frontend where I want to decompress it. The trouble I'm having is with converting the string to a Buffer because I believe the lz4 compressed file contains non-utf8 characters and so I'm unable to create a Uint8Array or a Buffer from it without converting non-utf8 characters to U+FFFD.
Some of the errors I'm seeing:
Error: Invalid magic number
Error: Invalid stream descriptor checksum
RangeError: invalid array length
Does anyone know how to get around this? Is there something simple I'm missing? Thanks in advance!
When I asked this question, I was convinced there was something wrong with the way I was converting things because of the following:
const sample = 'abcdefghijkl';
console.log('sample:', sample);
const buffer = Buffer.from(sample);
console.log('buffer', buffer);
const compressedSample = lz4.encode(sample);
console.log('compressedSample:', compressedSample);
const thisWorks = lz4.decode(compressedSample).toString();
// This works
console.log('thisWorks', thisWorks);
const compressedSampleString = compressedSample.toString();
console.log('compressedSample string:\n', compressedSampleString);
const bufferCompressed = Buffer.from(compressedSampleString);
console.log('bufferCompressed:', bufferCompressed);
console.log('bufferCompressed string:\n', bufferCompressed.toString());
// This doesn't...
const decompressedSample = lz4.decode(bufferCompressed);
But it turns out that axios was the issue as decribed here. If I replace axios.get with axios.request and set it to accept binary data, then decoding succeeds:
public async binaryGet(url: string): Promise<any> {
try {
const response = await axios.request({
method: 'GET',
url: `${this.config.api_domain}${url}`,
responseType: 'arraybuffer',
responseEncoding: 'binary',
});
return response.data;
} catch (error) {
this.throwAPIError(error);
}
}