I have a custom gRPC backend deployed behind an Endpoints Service Proxy (ESP) connected to Google Cloud Endpoints.
When sending a request with the X-Cloud-Trace-Context header set, I can see the spans recorded by ESP show up in my Stackdriver Trace dashboard.
However, my service is also sending requests to Google Cloud KMS as part of handling that request. I'd like Google Cloud to create trace spans for those sub-requests automatically for me as well; however, attaching the X-Cloud-Trace-Context header that ESP forwarded to me to the sub-requests sent to Cloud KMS does not cause any spans for those sub-requests to show up in Stackdriver Trace. The service account used to connect to Cloud KMS does have the "Stackdriver Trace Agent" role enabled.
Is it possible to tell Google Cloud services (such as Cloud KMS) to automatically generate trace spans for the current request's trace context, or do I need to manually generate traces for these requests in my backend code?
Cloud Trace doesn't currently generate service-side traces for requests to most GCP services, although we're aware of it as a valuable feature. To track how much of your latency is being consumed by KMS (or other services) you can create a client-side trace record using OpenCensus (Github) or similar.
Cloud KMS (as of this writing) doesn't support gRPC, but we are working on it.