rust tracing + opentelemetry stdout layer

I am building a rust API and experimenting with opentelemetry. Before I switch from my current tracing_subscriber custom layers to using an opentelemetry layer i wanted to see what the traces generated by opentelemetry looked like, so I am trying to set up a basic opentelemetry layer that uses an stdout exported to print out the traces to the console.

Unfortunately I can't seem to get the layer to print anything to console. Here is my setup.

Cargo.toml

...
tracing = "0.1"
tracing-subscriber = "0.3"
tracing-opentelemetry = "0.23"
opentelemetry_sdk = { version = "0.22", features = ["rt-tokio", "trace"] }
opentelemetry = { version = "0.22", features = ["trace"] }
opentelemetry-stdout = { version = "0.3", features = ["trace"] }

Here is my main

... some other imports
use opentelemetry::trace::{Tracer, TracerProvider as _};
use opentelemetry_sdk::export::trace::{ExportResult, SpanData, SpanExporter};
use opentelemetry_sdk::runtime;
use opentelemetry_sdk::trace::TracerProvider;
use tracing::info;
use tracing_subscriber::{layer::SubscriberExt, util::SubscriberInitExt, EnvFilter};

#[tokio::main]
async fn main() {
    // Set up application wide tracing
    setup_tracing().await;
    // Setting up the server etc. etc.
}

async fn setup_tracing() {
    let provider = TracerProvider::builder()
        .with_simple_exporter(opentelemetry_stdout::SpanExporter::default())
        .build();

    let tracer = provider.tracer("example");
    let telemetry = tracing_opentelemetry::layer().with_tracer(tracer);

    tracing_subscriber::registry()
        .with(EnvFilter::from_default_env())
        .with(StdoutLayer::new())
        .with(LogLayer::new().await)
        .with(MetricLayer::new())
        .with(telemetry)
        .init();
}

and here is an example of an instrumented axum route

#[instrument]
pub async fn get(Path(id): Path<u64>) -> Result<Json<Model>, Error> {
    info!("hello");
    let result = doctor::get(id).await?;
    Ok(JsonResponse(result))
}

When i call the get api I would expect to see an output from the telemetry layer, but none is outputted. I do see on the other hand an output form my custom layers as expected.

My code to build the telemetry stdout layer is pretty much identical to the example given in the tracing_opentelemetry crate as far as i can tell.

Anything else i am missing? My next step was to create a custom exporter or even tracer so i can verify if anything is even getting to them.

Solution

I have found the issue after looking at a similar question for opentelemetry-jaeger.

The problem is that the provider gets dropped when the setup_tracing function returns. The provider contains the opentelemetry processor, so if it gets dropped none of the traces get to the exporter.

It's important that the tracer provider stays around for the duration of the program. So the setup_tracing function should look like this

async fn setup_tracing() -> TracerProvider {
    let provider = TracerProvider::builder()
        .with_simple_exporter(opentelemetry_stdout::SpanExporter::default())
        .build();
    let tracer = provider.tracer("my-service");

    Registry::default()
        .with(EnvFilter::from_default_env())
        .with(StdoutLayer::new())
        .with(LogLayer::new().await)
        .with(MetricLayer::new())
        .with(OpenTelemetryLayer::new(tracer))
        .init();

    provider
}

and it can be called as such from main

let _provider = setup_tracing().await;

A bit of an obscure side effect that should be documented in the opentelemetry crates examples.