I am creating a c++ implementation of HDFS, using ONCRPC and Google Protobuf. The issue I'm facing is that I'm sending a protobuf object with multiple fields populated (sending the serialized string, parsing from it at the receving end), however, at the receiving end it erroneously says that the one of the fields has not been set/does not exist.
This is a part of my hdfs.proto file :
message AssignBlockRequest {
optional int32 handle = 1; // obtain using call to OpenFile
}
message AssignBlockResponse {
optional int32 status = 1;
optional BlockLocations newBlock = 2;
}
message BlockLocations {
optional int32 blockNumber = 1;
repeated DataNodeLocation locations = 2;
}
message DataNodeLocation {
optional string ip = 1;
optional int32 port = 2;
}
I'm using this in the "client" application to query the "namenode server" for a new block and a list of datanodelocations to which it can send data to write.
So, in my client :
AssignBlockResponse assignnewblock_ ( int fhandle, CLIENT* clnt ) {
AssignBlockRequest req;
req.set_handle(fhandle);
//send request to nn
string str;
req.SerializeToString(&str);
static char *cstr = new char[str.length() + 1];
memcpy(cstr, str.c_str(), str.length()+1);
char **result_abreq;
result_abreq = assignblock_1( &cstr, clnt );
//handle response
AssignBlockResponse rsp;
string str_arg (*result_abreq);
rsp.ParseFromString(str_arg);
cout << "NN RETURNED : " << rsp.status() << " " << rsp.has_newblock() << endl;
return rsp;
}
while in my namenode server.cc
char **
assignblock_1_svc(char **argp, struct svc_req *rqstp)
{
AssignBlockRequest req;
string str_arg (*argp);
req.ParseFromString(str_arg);
AssignBlockResponse rsp;
if ( DataNodeList.empty() ) { // no DN available
rsp.set_status (1);
}
else {
rsp.set_status (0);
int BL_NUM = 0;
vector<int> shuf;
BlockLocations bl;// = new BlockLocations;
bl.set_blocknumber(BL_NUM);
rsp.mutable_newblock()->CopyFrom(bl);
}
cout << "NN RETURNED : " << rsp.status() << " " << rsp.has_newblock() << endl;
string str;
rsp.SerializeToString(&str);
static char *cstr = new char[str.length() + 1];
memcpy(cstr, str.c_str(), str.length()+1);
return &cstr;
}
The NN outputs "0 1" while the client upon receiving this AssignBlockResponse type request shows "0 0" i.e. it gets the status right (tested by varying the status set in the AssignBlockResponse message), but never detects the "newblock" field sent by the server.cc to it.
Any help would be greatly appreciated.
-- EDIT 1 --
Protocol buffer serializing with inheritance. Derived classes are empty
This may be of interest. I still can't get my code to work, howvever.
I've come across this in my early work with protocol buffers.
Don't serializeToString. serializeToArray having first built a vector big enough (call ByteSize() on the Message)
The problem is that your serialised byte stream contains a zero byte, which is interpreted as an end-of-string when converting the char* to a string.
This means you end up parsing an incomplete message, hence missing fields.