javacompressionzlibdeflateinflate

Java.util.zip.DataFormatException: incorrect header check


First post, usually I find what Im looking for in other threads but not this time:

Im using javas Deflater and Inflater to compress/ decompress some data I send between a server and client application that Im working on.

It works just fine for 99% of my tests. However there is one particular dataset that when inflated throws this exception from the inflater.inflate() method:

DataFormatException: incorrect header check

There is nothing special about the data compared to the other runs. Its just a bunch of numbers seperated by commas "encoded" as a String and then done .getBytes() to. The only thing I know is that its a bit bigger this time. There is not encoding happening anywhere between the compression -> decompression steps.


This is the code to send something to either the client or the server. The code is shared.

OutputStream outputStream = new DataOutputStream(socket.getOutputStream());
byte[] uncompressed = SOMEJSON.toString().getBytes();
int realLength = uncompressed.length;

// compress data
byte[] compressedData = ByteCompression.compress(uncompressed);
int compressedLength = compressedData.length;
    
outputStream.write(ByteBuffer.allocate(Integer.BYTES).putInt(compressedLength).array());
outputStream.write(ByteBuffer.allocate(Integer.BYTES).putInt(realLength).array());
    
outputStream.write(compressedData);
outputStream.flush();

This is the code to receive data (either client or server) also shared:

DataInputStream dataIn = new DataInputStream(socket.getInputStream());
int compressedLength = dataIn.readInt();
int realLength = dataIn.readInt();

errorhandling.info("Packet Reader", "Expecting " + compressedLength + " (" + realLength + ") bytes.");
byte[] compressedData = new byte[compressedLength];
            
int readBytes = 0;
while (readBytes < compressedLength) {
    int newByteAmount = dataIn.read(compressedData);
                
    // catch nothing being read or end of line
    if (newByteAmount <= 0) {
        break;
    }
    readBytes += newByteAmount;
}
            
if (readBytes != compressedLength) {
    errorhandling.info("Packet Reader", "Read byte amount differs from expected bytes.");
    return new ErrorPacket("Read byte amount differs from expected bytes.").create();
}

byte[] uncompressedData = ByteCompression.decompress(compressedData, realLength);
String packetData = new String(uncompressedData);

Here are the methods to compress and decompress a byteArray (you guessed right its shared):

public static byte[] compress(byte[] uncompressed) {
        Deflater deflater = new Deflater(Deflater.BEST_COMPRESSION);
        deflater.setInput(uncompressed);
        deflater.finish();

        byte[] compressed = new byte[uncompressed.length];
        int compressedSize = 0;
        while (!deflater.finished()) {
            compressedSize += deflater.deflate(compressed);
        }
        
        deflater.end();

        return Arrays.copyOfRange(compressed, 0, compressedSize);
    }
    
    public static byte[] decompress(byte[] compressed, int realLength) throws DataFormatException {     
        Inflater inflater = new Inflater(true);
        inflater.setInput(compressed);

        byte[] uncompressed = new byte[realLength];
        while (!inflater.finished()) {
            inflater.inflate(uncompressed); // throws DataFormatException: incorrect header check (but only super rarely)
        }
        inflater.end();

        return uncompressed;
    }

So far Ive tried differnt compression levels and messing with the "nowrap" option for both Deflater and Inflater (all combinations):

// [...]
Deflater deflater = new Deflater(Deflater.BEST_COMPRESSION, true);
// [...]
Inflater inflater = new Inflater(true);

But that just results in these exceptions (but again only for that one particulat dataset):

DataFormatException: invalid stored block lengths
DataFormatException: invalid distance code

Im sorry for this wall of text but at this point I really dont know anymore what could be causing this issue.


Solution

  • Alright here is the solution:

    My assumption was that this loop would APPEND new read data to the byte array where it last stopped THIS IS NOT THE CASE (it seems to stop reading after 2^16 bytes so thats why I dont get this issue with smaller packets).

    This is wrong:

    int readBytes = 0;
    while (readBytes < compressedLength) {
        int newByteAmount = dataIn.read(compressedData); // focus here!
        readBytes += newByteAmount;
    }
    

    So whats happening is that the data is read correctly however the output array is overwriting itself!! Thats why I see wrong data at the start and a bunch of 00 00 at the end (because it never actually reached that part of the array)!

    Using this instead fixed my issue:

    dataIn.readFully(compressedData);
    

    What concerns me is that I see the first variant of the code A LOT. Thats what I found when googling it.