c++jsonglibcasiolz4

"corrupted size vs. prev_size" when using LZ4


I'm using JSON to pass messages between a server and a client, both written in C++. To get a performance boost, I'm evaluating compression, as well as binary formats (like MessagePack). I started with compression first, as it required the least amount of code changes. I should note that before I added compression, the program works as expected (at least on the inputs I use for testing). Here is the code both the client and server use for communication:

void serializeInt(std::array<uint8_t,8>& buf, int i, uint32_t val) { //Assumes that val is a 32-bit number (almost always true). Serializes in little endian in endian-agnostic way
    buf[i+0] = (val) & 0xFF;
    buf[i+1] = (val >> 8) & 0xFF;
    buf[i+2] = (val >> 16) & 0xFF;
    buf[i+3] = (val >> 24) & 0xFF;
}

uint32_t deserializeInt(std::array<uint8_t,8>& buf, int i){ //Deserialzes from little endian in endian-agnostic way
    return buf[i+0] | (buf[i+1] << 8) | (buf[i+2] << 16) | (buf[i+3] << 24);
}

boost::json::object readFromConn(){
    auto curr=currStruct();
    asio::error_code ec;
    
    asio::read(*(curr->conn), asio::buffer(curr->size_buf, 8), asio::transfer_exactly(8), ec);
    if (ec){
        throw RWError(ec);
    }
    
    auto compressed_size=deserializeInt(curr->size_buf, 0);
    auto input_size=deserializeInt(curr->size_buf, 4);
    

    auto compressed_data=(char*)malloc(compressed_size);
    auto input=(char*)malloc(input_size);
    
    asio::read(*(curr->conn), asio::buffer(compressed_data, compressed_size), asio::transfer_exactly(compressed_size), ec);
    
    if (ec){
        throw RWError(ec);
    }
    
    auto line=std::string_view(input,input_size);
    LZ4_decompress_safe(compressed_data, input, compressed_size, input_size);

    boost::json::object json=boost::json::parse(line,{}, {.max_depth=180,.allow_invalid_utf8=true,.allow_infinity_and_nan=true}).get_object();
    
    free(input);
    free(compressed_data);
    
    return json;
}

void writeToConn(boost::json::object& json){
    auto curr=currStruct();
    
    json["uuid"]=uuid;
    
    auto line=boost::json::serialize(json,boost::json::serialize_options{.allow_infinity_and_nan=true});

    auto input_size=line.size();
    auto input=line.c_str();
    auto max_compressed_size=LZ4_compressBound(input_size);
    auto compressed_data=(char*)malloc(max_compressed_size);
    
    auto compressed_size=LZ4_compress_default(input, compressed_data, input_size, max_compressed_size);
    
    serializeInt(curr->size_buf, 0, compressed_size);
    serializeInt(curr->size_buf, 4, input_size);
    asio::error_code ec;
    asio::write(*(curr->conn), asio::buffer(curr->size_buf,8), ec);
    
    if (ec){
        throw RWError(ec);
    }
    
    asio::write(*(curr->conn), asio::buffer(compressed_data,compressed_size), ec);
    
    if (ec){
        throw RWError(ec);
    }
    
    free(compressed_data);
}

However, once I added compression, I kept getting "corrupted size vs. prev_size" errors, but not in the server code.

I copied the compression code from LZ4's simple_buffer.c file, so assuming it works, I'm at a loss at the issue (maybe I overlooked something subtle?)


Solution

  • The issue turned out to have nothing to do with the snippet I posted, but with some other file: I updated the definition of a struct in the header, but forgot to rebuild the file that used the struct, so the code tried to access fields that didn't exist.