google-cloud-pubsubspring-cloud-gcp

Spring Cloud GCP PubSub Tracing using OpenTelemetry vs Spring Cloud Trace

I'm trying to trace PubSub messaging across different Spring microservices. We are running two Spring microservices on GKE. Service 1 publishes to PubSub and Service 2 subscribes to the same topic. We use the spring-cloud-gcp-trace lib for Tracing in our services.

I see two approaches to trace PubSubMessages

  1. OpenTelemetry Tracing in the Publisher & Subscriber https://cloud.google.com/pubsub/docs/open-telemetry-tracing
  2. Using TracePubSubAutoConfiguration with spring-cloud-gcp-trace (https://github.com/GoogleCloudPlatform/spring-cloud-gcp/tree/main/spring-cloud-gcp-autoconfigure/src/main/java/com/google/cloud/spring/autoconfigure/trace/pubsub)

There are no examples for the second option. With option 1, I see examples where I can trace the message from the publisher to subscriber with time spent in the client libraries. But with option 2 I'm not sure if we will be able to do that. As the spring-cloud-gcp-trace uses Brave instrumentation and not necessarily OpenTelemetry. The Publisher and Subscriber API requires OpenTelemetrySdk

The questions are

  1. Are these two different things and can coexist?
  2. Can I use the trace lib to get the kind of tracing I would get from the open telemetry option?

Thanks

Solution

  • The OpenTelemetry instrumentation for the Pub/Sub Java client is implemented within the com.google.cloud.pubsub.v1.Publisher and com.google.cloud.pubsub.v1.Subscriber classes, which the Pub/Sub Spring client is built on top of. It is a separate implementation from TracePubSubAutoConfiguration with spring-cloud-gcp-trace and thus both the OpenTelemetry and Brave instrumentations can be used at the same time.

    Since the OpenTelemetry instrumentation is implemented at a lower level (the actual Publisher and Subscriber client classes as opposed to a wrapper of them), it allows for more a detailed trace of your messages within Pub/Sub, including flow control, batching, and lease management. This same level of detail is not possible when using just the spring-cloud-gcp-trace Brave instrumentation.

    In order to enable OpenTelemetry instrumentation for a Pub/Sub Spring client, you need to will need to implement a custom PublisherFactory that instantiates an OpenTelemetry instance with the required tracers, exporters, etc. The sample for the Pub/Sub Java client can be used as an example for this.