goprotocol-buffersgrpcprotobuf-go

protoc Go Package Command Local Overwrite?

Following the Quick Start gRPC Go guide on the official gRPC website it has a step which asks the user to recompile the updated .proto file using this command:

$ protoc --go_out=. --go_opt=paths=source_relative --go-grpc_out=. --go-grpc_opt=paths=source_relative helloworld/helloworld.proto

I'm a little confused as to how the newly compiled protobuf files are consumed by the "human written" Go code.

In the example "human written" Go code they reference the protobuf code using the following import:

pb "google.golang.org/grpc/examples/helloworld/helloworld"

protoc does not update this package but instead updates the helloworld/helloworld.proto in the same directory that the command is run in. How does the protoc command ensure that the "human written" Go code consumes the newly compiled protobuf code?

Solution

  • I find this one of the more confusing aspects of Protobufs (and gRPC).

    I think the issue being solved is that Protobufs need to:

    1. Permit namespacing to scope services|messages to e.g. DNS domains
    2. Support a myriad programming languages (which implement namespacing differently).

    protochas options that permit (re)mapping of e.g. a protobuf's package v1/api to a language-specific namespace using protobuf options, e.g. go_package). See Go Generated Code for Golang.

    This is slightly more complex when using Go Modules but, in summary, what you're experiencing is probably some (unexpected) combination of the above where, the sample code assumes one module name and protoc is building on the assumption of a different one.

    TL;DR update the code's module reference to reflect the correctly generated pb path. If the generate code is in the wrong place, you can simply move it to the correct subdirectory (path) but it's better to update your protoc command to generate the files to the correct directory.

    Example

    something.proto:

    syntax = "proto3";

package v1alpha1;

option go_package = "github.com/me/my-protos;v1alpha1";

    NOTE go_package aliases the proto package v1alpha1 to what I want to reference as github.com/me/my-protos in Golang.

    Then I generate with:

    MODULE="github.com/me/my-protos"
protoc \
--proto_path=. \
--go_out=./api/v1alpha1 \
--go_opt=module=${MODULE} \
--go-grpc_out=./api/v1alpha1 \
--go-grpc_opt=module=${MODULE} \
./something.proto

    NOTE This example generate gRPC code too. It avoids (!) protoc creating a path github.com/me/my-protos for the generated code because I'm generating the source in that repo. I just want the relative path ./api/v1alpha where the files will be created.

    Which yields:

    my-protos
├── something.proto
├── api
│   └── v1alpha1
│       ├── something_grpc.pb.go
│       └── something.pb.go
├── go.mod
├── go.sum
└── README.md

    And I can import with:

    import (
    pb "github.com/me/my-protos/api/v1alpha1"
)

    NOTE From a different repo, I can now access my protos repo github.com/me/my-project/api/v1alpha1 combining the repro and the generated location to give my desired path.