[SOLVED] Protobuf message - parsing difference between binary and text files

Protobuf message - parsing difference between binary and text files

During my implementation on a protocol buffer application, I tried to work with the text pbtxt files to ease up my programming. The idea was to switch to the pb binary format afterward, once I had a clearer understanding of the API. (I am working in C++)

I made my application working by importing the file with TextFormat::Parse. (The content of the file came from TextFormat::Print). I then generated the corresponding binary file, that I tried to import with myMessageVariable.ParsefromCodedStream (file not compressed). But I notice that only a very small part of the message is imported. The myMessageVariable.IsInitialized returns true, thus I guess that the library "thinks" that it has completely imported the file.

So my question: is there something different in the way the file are imported that could make the import "half-fail"? (Besides the obvious reason that one is binary and the other one is in text?) And what can we do against it?

Solution

There are a few differences in reading text data and reading binary data:

Text files sometimes use automatic linefeed conversion (\r\n vs. \n), especially on Windows platforms. This has to be disabled by opening the file in binary mode.
Binary files can contain null bytes at any point. Some text processing functions stop reading at the first null byte.

It could help if you can determine more about how much of the message gets parsed. Then you can look at what kind of bytes are near the problem point, using e.g. hex editor.