google-cloud-platformprotocol-buffersgoogle-cloud-pubsubgoogle-schemas

Google Cloud Pub/Sub Schemas - Too many protobuf messages in single schema


I'm defining a schema for a pub/sub topic.

I have 7 proto files, and I'm able to combine them. When I did this, GCP outputted that there were too many protobuf messages in a single schema. I believe this is something to do with the 50Kb schema definition limit. I have looked at the related stack thread, but I am not able to combine my messages within other messages since they need to be used in multiple other places.

If I was to make 1 schema for each of my protobuf files, would I be able to use multiple schemas for a single topic? Or would I have to combine them (which has thrown me the above error).


Solution

  • Pub/Sub proto schemas can only contain a single top-level message definition. In other words, a schema such as the following would return the error you see:

    message M1 {
      string f1 = 1;
    }
    
    messages M2 {
      M1 m1 = 1;
    }
    

    In such a case, you'd need to nest the definitions:

    message M2 {
      message M1 {
        string f1 = 1;
      }
      M1 m1 = 1;
    }
    

    This is noted in the Pub/Sub schema documentation. It also mentions that imports are not allowed, which is why you are getting the error about trying to import the timestamp proto.

    A topic can only only have a single schema associated with it, so you cannot separate these into multiple schemas and use them all in the same topic. You could consider defining a single schema that has a union of the types you wish to use:

    message FullSchema {
      message M1 {
        string f1 = 1;
      }
    
      message M2 {
        string f1 = 1;
      }
    
      oneof value {
        M1 m1 = 1;
        M2 m2 = 2;
      }
    }