I'm writing a program in C++, on windows platform. I would like to compress some data storing in a char[] array, and output it to a file, and later I will upload the file to a unix server and I want it can be decompressed by gzip -d.
After a lot of research, I choose miniz. Also, I found the gzip file format here.
Here is the code snippet of creating gzip file: (sorry I didn't put the definition of some variables; they're defined somewhere else)
unsigned long zsize;
zpkg[0] = 0x1F;
zpkg[1] = 0x8B;
zpkg[2] = 8;
zpkg[3] = 0;
zpkg[4] = 0;
zpkg[5] = 0;
zpkg[6] = 0;
zpkg[7] = 0;
zpkg[8] = 0;
zpkg[9] = 0xFF;
compress2(zpkg + 10, &zsize, pkg, pkgSize, MZ_DEFAULT_LEVEL);
int footerStart = (int)zsize + 10;
mz_ulong crc = mz_crc32(MZ_CRC32_INIT, zpkg + 10, zsize);
zpkg[footerStart] = crc & 0xFF;
zpkg[footerStart + 1] = (crc >> 8) & 0xFF;
zpkg[footerStart + 2] = (crc >> 16) & 0xFF;
zpkg[footerStart + 3] = (crc >> 24) & 0xFF;
zpkg[footerStart + 4] = pkgSize & 0xFF;
zpkg[footerStart + 5] = (pkgSize >> 8) & 0xFF;
zpkg[footerStart + 6] = (pkgSize >> 16) & 0xFF;
zpkg[footerStart + 7] = (pkgSize >> 24) & 0xFF;
And later just output the zpkg array to a file. However this doesn't work; when I decompress it with gzip, the error messsage is:
gzip: data stream error
gzip: test.gz: uncompress failed
Can anybody please point me out what I'm doing wrong?
Thanks for Mark Adler and Michael, I figure out a working solution.
First, as Mark pointed out, I should make miniz return a raw deflate data stream. This can be done by passing -MZ_DEFAULT_WINDOW_BITS (notice the minus sign) to mz_deflateInit2() as the fourth parameter. Looking into the miniz source code, the compress2() function eventually calls the mz_deflateInit2() with MZ_DEFAULT_WINDOW_BITS, which means adding zlib header and footer. So the easiest fix is adding a minus sign there, so that I can still use the compress2() function. (this works for me because I only call this function at one place)
Second, as Michael pointed out, the CRC code should be calculated on uncompressed data. So I fix it like this:
mz_ulong crc = mz_crc32(MZ_CRC32_INIT, pkg, pkgSize);
After making the two changes above, gzip -d doesn't complain anymore.
compress2() produces a zlib stream, which is deflate compressed data with a zlib header and trailer. For what you are doing, you want just the raw deflate compressed stream to stick in your manually generated gzip header and trailer.
You could either: a) discard the first two and last four bytes of the output of compress2() to strip the zlib header and trailer, b) use deflateInit2(), deflate(), and deflateEnd() instead of compress2() and select the raw deflate format, or c) use those same functions and instead select the gzip format, and get rid of your manually constructed gzip header and trailer since deflate() will do that for you.
I recommend c).