I'm using JSON to pass messages between a server and a client, both written in C++. To get a performance boost, I'm evaluating compression, as well as binary formats (like MessagePack). I started with compression first, as it required the least amount of code changes. I should note that before I added compression, the program works as expected (at least on the inputs I use for testing). Here is the code both the client and server use for communication:
void serializeInt(std::array<uint8_t,8>& buf, int i, uint32_t val) { //Assumes that val is a 32-bit number (almost always true). Serializes in little endian in endian-agnostic way
buf[i+0] = (val) & 0xFF;
buf[i+1] = (val >> 8) & 0xFF;
buf[i+2] = (val >> 16) & 0xFF;
buf[i+3] = (val >> 24) & 0xFF;
}
uint32_t deserializeInt(std::array<uint8_t,8>& buf, int i){ //Deserialzes from little endian in endian-agnostic way
return buf[i+0] | (buf[i+1] << 8) | (buf[i+2] << 16) | (buf[i+3] << 24);
}
boost::json::object readFromConn(){
auto curr=currStruct();
asio::error_code ec;
asio::read(*(curr->conn), asio::buffer(curr->size_buf, 8), asio::transfer_exactly(8), ec);
if (ec){
throw RWError(ec);
}
auto compressed_size=deserializeInt(curr->size_buf, 0);
auto input_size=deserializeInt(curr->size_buf, 4);
auto compressed_data=(char*)malloc(compressed_size);
auto input=(char*)malloc(input_size);
asio::read(*(curr->conn), asio::buffer(compressed_data, compressed_size), asio::transfer_exactly(compressed_size), ec);
if (ec){
throw RWError(ec);
}
auto line=std::string_view(input,input_size);
LZ4_decompress_safe(compressed_data, input, compressed_size, input_size);
boost::json::object json=boost::json::parse(line,{}, {.max_depth=180,.allow_invalid_utf8=true,.allow_infinity_and_nan=true}).get_object();
free(input);
free(compressed_data);
return json;
}
void writeToConn(boost::json::object& json){
auto curr=currStruct();
json["uuid"]=uuid;
auto line=boost::json::serialize(json,boost::json::serialize_options{.allow_infinity_and_nan=true});
auto input_size=line.size();
auto input=line.c_str();
auto max_compressed_size=LZ4_compressBound(input_size);
auto compressed_data=(char*)malloc(max_compressed_size);
auto compressed_size=LZ4_compress_default(input, compressed_data, input_size, max_compressed_size);
serializeInt(curr->size_buf, 0, compressed_size);
serializeInt(curr->size_buf, 4, input_size);
asio::error_code ec;
asio::write(*(curr->conn), asio::buffer(curr->size_buf,8), ec);
if (ec){
throw RWError(ec);
}
asio::write(*(curr->conn), asio::buffer(compressed_data,compressed_size), ec);
if (ec){
throw RWError(ec);
}
free(compressed_data);
}
However, once I added compression, I kept getting "corrupted size vs. prev_size" errors, but not in the server code.
I copied the compression code from LZ4's simple_buffer.c
file, so assuming it works, I'm at a loss at the issue (maybe I overlooked something subtle?)
The issue turned out to have nothing to do with the snippet I posted, but with some other file: I updated the definition of a struct in the header, but forgot to rebuild the file that used the struct, so the code tried to access fields that didn't exist.