OpenTelemetry span exporter blocks from within tonic RPC

I want to export spans using the OpenTelemetry Protocol from inside of a tonic service. My attempts to do this seemingly result in deadlock. I've created a minimal example here. Here's a summary:

I first create the otlp tracer and integrate it with the tracing crate:

let tracer = opentelemetry_otlp::new_pipeline()
    .tracing()
    .with_exporter(opentelemetry_otlp::new_exporter().tonic())
    .install_simple()
    .unwrap();
let telemetry = tracing_opentelemetry::layer().with_tracer(tracer);
let subscriber = tracing_subscriber::Registry::default().with(telemetry);
tracing::subscriber::set_global_default(subscriber).unwrap();

At this point, creating a tracing span and emitting an event works. I can receive and view it using this utility.

Next, I start the tonic server:

tonic::transport::Server::builder()
    .add_service(greeter_server::GreeterServer::new(GreeterService {}))
    .serve("[::1]:50051".parse().unwrap())
    .await
    .unwrap();

The RPC method is decorated with tracing::instrument, and attempts to emit an event:

#[tonic::async_trait]
impl greeter_server::Greeter for GreeterService {
    #[tracing::instrument]
    async fn say_hello(...) -> ... {
        tracing::info!("hello from server");
        ...
    }
}

When I run the client, the server thread servicing the request blocks on emitting the event indefinitely.

It appears to be blocking in opentelemetry_sdk::trace::span_processor::SimpleSpanProcessor::on_end in this statement:

let result = self
    .exporter
    .lock()
    .map_err(|_| TraceError::Other("SimpleSpanProcessor mutex poison".into()))
    .and_then(|mut exporter| futures_executor::block_on(exporter.export(vec![span])));

Where am I going wrong?

The solution is to use the OpenTelemetry Tokio batch exporter instead of the simple exporter:

let tracer = opentelemetry_otlp::new_pipeline()
    .tracing()
    .with_exporter(exporter)
    .install_batch(opentelemetry_sdk::runtime::Tokio)
    .unwrap();

This requires the following feature:

opentelemetry_sdk = { version = "0.23.0", features = ["rt-tokio"] }

Credit to https://github.com/denizenging over on the Tokio Discord server for figuring this out.