I see istio is adding x-b3-traceid, x-b3-spanid and other headers to the incoming request when tracing is enabled. But none of them are returned to the caller.
I am able to capture the x-b3-traceid in the log and can find it out in Tempo/Grafana.
I can see the traceid at the istio envoy proxy (sidecar), I am able to access the header using EnvoyFilter.
Can someone let me know where it is filtered?
TL;DR
these are the headers so that jaeger or zipkin can track individual requests. This application is responsible for their proper propagation. Additionally, they are request headers and not response header, so everything works fine.
Explanation:
It looks okay. First, let's start with what these requests are:
Field Name Request/ Response Type Description
x-request-id request The x-request-idheader is used by Envoy to uniquely identify a request as well as perform stable access logging and tracing
x-b3-traceid request The x-b3-traceidHTTP header is used by the Zipkin tracer in Envoy. The TraceId is 64-bit in length and indicates the overall ID of the trace. Every span in a trace shares this ID
x-b3-spanid request The x-b3-spanidHTTP header is used by the Zipkin tracer in Envoy. The SpanId is 64-bit in length and indicates the position of the current operation in the trace tree
x-b3-sampled request The x-b3-sampledHTTP header is used by the Zipkin tracer in Envoy. When the Sampled flag is either not specified or set to 1, the span will be reported to the tracing system
On the github you can find the question: What is the usage of X-B3-TraceId, traceId, parentSpanId and spanId ?:
http headers contain only a small amount of data: IDs and sampling flags these go synchronously with your application requests, and allow the other side to continue the trace. https://github.com/openzipkin/b3-propagation#overall-process If zipkin is enabled, details including these IDs and other data like duration send asynchronously to zipkin after a request completes. you can see a diagram about that here under "Example Flow" http://zipkin.io/pages/architecture.html
X-B3-TraceId is same or different from every call of the same client? different per overall request. each top-level request into your system has
a different trace id. Each step in that request has a different span id
X-B3-SpanId is not send back to the caller, then how could i set the parent(which show be the X-B3-SpanId of the the present call) of the next call? Here is a response shows the absent of X-B3-SpanId in the header： I don't quite understand. The parent is only used when creating a span.
The span is created before a request is sent. So, there's no relevance to response headers in span creation.
In this doc you can find information about headers from istio site:
Distributed tracing enables users to track a request through mesh that is distributed across multiple services. This allows a deeper understanding about request latency, serialization and parallelism via visualization.
Istio leverages Envoy’s distributed tracing feature to provide tracing integration out of the box. Specifically, Istio provides options to install various tracing backend and configure proxies to send trace spans to them automatically. See Zipkin, Jaeger and Lightstep task docs about how Istio works with those tracing systems.
If you want to full understand how it works you should read this envoyproxy documentation:
Distributed tracing allows developers to obtain visualizations of call flows in large service oriented architectures. It can be invaluable in understanding serialization, parallelism, and sources of latency. Envoy supports three features related to system wide tracing:
Request ID generation: Envoy will generate UUIDs when needed and populate the x-request-id HTTP header. Applications can forward the x-request-id header for unified logging as well as tracing.
Client trace ID joining: The x-client-trace-id header can be used to join untrusted request IDs to the trusted internal x-request-id.
External trace service integration: Envoy supports pluggable external trace visualization providers, that are divided into two subgroups:
External tracers which are part of the Envoy code base, like LightStep, Zipkin or any Zipkin compatible backends (e.g. Jaeger), and Datadog.
External tracers which come as a third party plugin, like Instana.
Answering your question:
Can someone let me know where it is filtered?
They do this by default. It is the application (zipkin, jaeger) that is responsible for acting on these headers. Additionally, they are request headers and not response header, so everything works fine.