pythongoogle-app-engineprotocol-buffersprotorpc

Converting proto buffer to ProtoRPC

In a Python script, mylibrary.py, I use Protocol Buffers to model data using the following approach:

I want to implement Cloud Endpoints Framework on App Engine that imports and uses the aforementioned Python script, however Cloud Endpoints uses ProtoRPC, not 'standard' Protocol Buffers.

My App Engine Python module, main.py, imports from protorpc rather than using the 'offline' protoc compiler to generate serialization and deserialization code:

from protorpc import messages
from protorpc import remote

Messages are not defined​ using .proto files. Instead, classes are defined, inheriting from protorpc.messages.Message:

class MyMessageDefinition(messages.Message)

Can Proto Buffers be converted to Proto RPC equivalents? I don't really want to change mylibrary.py to use ProtoRPC, since it's less generic than Protocol Buffers.

Solution

  • After eight months and lots experimentation, I'll add my opinion. I hope it saves someone time.

    Choose Your Framework First

    There are different Cloud Endpoint offerings from Google Cloud. All can be used for JSON/REST APIs. This wasn't immediately clear to me. Cloud Endpoints is a very high-level phrase covering development, deployment and management of APIs on multiple Google Cloud backends.

    The point here is that after deciding to use Cloud Endpoints, you must still decide on backend technologies to serve your API. The documentation feels a little hidden away, but I strongly recommend starting with the Google Cloud Endpoints doc.

    You can choose between:

    1. OpenAPI Specification
    2. Endpoints Frameworks
    3. gRPC

    Choose Your Implementation Second

    Within each API Framework there’s a choice of Cloud implementations upon which your API (service) can run:

    OpenAPI Specification - for JSON/REST APIs implemented on:

    Endpoints Frameworks - for JSON/REST APIs implemented on:

    gRPC - for gRPC APIs implemented on:

    When posting question here, I was using Endpoints Frameworks running on Google App Engine standard environment with Python. I then migrated my API (service) to gRPC on Google Compute Engine.

    The observant among you may notice both the OpenAPI Specification and Endpoints Frameworks can be used for JSON/REST APIs, while gRPC only exposes a gRPC API. So how did I port my REST API from Endpoints Frameworks to gRPC? The answer is Transcoding HTTP/JSON to gRPC (which I learnt along the way, and was not immediately clear to me). So, don't rule out gRPC just because you want REST/HTTP.

    The Answer

    So how does this related to my original question?

    That I was trying to convert between .proto files and gRPC annotations at all, meant I had taken a wrong-turning along the way.

    If you want to write an application using plain .proto files, then choose gRPC on Compute Engine. If you need this to be a REST API, this can be done, but you'll need to add an ESP into your backend configuration. It's pretty much an NGINX sever setup as a reverse proxy. The only downside here is you'll need some Docker knowledge to ensure the ESP (proxy) and your gRPC server can communicate (Docker networking).

    If your code is already on an App Engine, and you want to expose it as a REST API with minimum effort and still get good API management features, choose Endpoints Frameworks. Warning: I moved away from this because it was prohibitively expensive (I was getting billed in the region of $100 USD monthly).

    If you want to avoid .protos altogether, then go with OpenAPI Specification.

    Lastly, if you want to offer programmatic integration, client libraries, or you want to offer a microservice, then really do consider gRPC. It's easy to remove the ESP (proxy) and run a gRPC server on nearly any machine (as long as the Protocol Buffer Runtime is installed.

    Ultimately I settled on gRPC on Compute Engine with Docker. I also have an ESP to provide a HTTP transcoding to gRPC and vice-versa. I like this approach for a few reasons:

    1. You learn a lot: Docker, Docker Networking, NGINX configuration, Protocol Buffers, ESP (Cloud Proxy), gRPC servers.
    2. The service (core business) logic can be written with plain-old gRPC. This allows the service to be run on any machine without a web server. Your business logic, is the server :)
    3. Protocol Buffers / gRPC are excellent for isolating business logic as a service...or microservice. They're also good for providing well-defined interfaces and libraries.

    Avoid These Mistakes