OS and protobuf version
go1.18.1 linux/amd64, github.com/golang/protobuf v1.5.2
Introduction
I am trying to use recursive proto definitions.
.proto file
message AsyncConsensus {
int32 sender = 1;
int32 receiver = 2;
string unique_id = 3; // to specify the fall back block id to which the vote asyn is for
int32 type = 4; // 1-propose, 2-vote, 3-timeout, 4-propose-async, 5-vote-async, 6-timeout-internal, 7-consensus-external-request, 8-consensus-external-response, 9-fallback-complete
string note = 5;
int32 v = 6 ; // view number
int32 r = 7;// round number
message Block {
string id = 1;
int32 v = 2 ; // view number
int32 r = 3;// round number
Block parent = 4;
repeated int32 commands = 5;
int32 level = 6; // for the fallback mode
}
Block blockHigh = 8;
Block blockNew = 9;
Block blockCommit = 10;
}
The following is how I Marshal and Un-Marshal
func (t *AsyncConsensus) Marshal(wire io.Writer) error {
data, err := proto.Marshal(t)
if err != nil {
return err
}
lengthWritten := len(data)
var b [8]byte
bs := b[:8]
binary.LittleEndian.PutUint64(bs, uint64(lengthWritten))
_, err = wire.Write(bs)
if err != nil {
return err
}
_, err = wire.Write(data)
if err != nil {
return err
}
return nil
}
func (t *AsyncConsensus) Unmarshal(wire io.Reader) error {
var b [8]byte
bs := b[:8]
_, err := io.ReadFull(wire, bs)
if err != nil {
return err
}
numBytes := binary.LittleEndian.Uint64(bs)
data := make([]byte, numBytes)
length, err := io.ReadFull(wire, data)
if err != nil {
return err
}
err = proto.Unmarshal(data[:length], t)
if err != nil {
return err
}
return nil
}
func (t *AsyncConsensus) New() Serializable {
return new(AsyncConsensus)
}
My expected outcome
When marshaled and sent to the same process via TCP, it should correctly unmarshal and produce correct data structures.
Resulting error
error "cannot parse invalid wire-format data"
Additional information
I tried with non-recursive .proto definitions, and never had this issue before.
This is not a bug with Protobuf, but its a mater of how you marshal and unmarshal protobuf structs.
As a concrete guideline, never concurrently marshal and unmarshal protobuf structs as it my lead to race conditions.
In the specific example you have provided, I see recursive data structs, so even if you use a separate struct for each invocation of marshal and unmarshal, it's likely that the pointers in the parent can lead to shared pointers.
Use a deep copy technique to remove any dependency so that you do not run in to race conditions.
func CloneMyStruct(orig *proto.AsyncConsensus_Block) (*proto.AsyncConsensus_Block, error) {
origJSON, err := json.Marshal(orig)
if err != nil {
return nil, err
}
clone := proto.AsyncConsensus_Block{}
if err = json.Unmarshal(origJSON, &clone); err != nil {
return nil, err
}
return &clone, nil
}