Rust OpenTelemetry OTLP w/ Honeycomb

I'm trying to use opentelemetry and open telemetry-otlp to provide observability data over OTLP to Honeycomb.

I'm using something like this as a proof of concept (extracted out into this repo if you want to run it: https://github.com/timfpark/honeycomb-rust-poc)

fn init_tracer(metadata: &MetadataMap) -> Result<sdktrace::Tracer, TraceError> {
    let opentelemetry_endpoint =
        env::var("OTEL_ENDPOINT").unwrap_or_else(|_| "https://api.honeycomb.io".to_owned());
    let opentelemetry_endpoint =
        Url::parse(&opentelemetry_endpoint).expect("OTEL_ENDPOINT is not a valid url");

    opentelemetry_otlp::new_pipeline()
        .tracing()
        .with_exporter(
            opentelemetry_otlp::new_exporter()
                .tonic()
                .with_endpoint(opentelemetry_endpoint.as_str())
                .with_metadata(metadata.clone())
                .with_tls_config(
                    ClientTlsConfig::new().domain_name(
                        opentelemetry_endpoint
                            .host_str()
                            .expect("OTEL_ENDPOINTshould have a valid host"),
                    ),
                ),
        )
        .install_batch(opentelemetry::runtime::Tokio)
}

#[tokio::main]
async fn main() -> anyhow::Result<()> {

    let mut metadata = MetadataMap::with_capacity(2);

    metadata.insert(
        "x-honeycomb-team",
        "...honeycomb api key...".parse().unwrap(),
    );

    metadata.insert("x-honeycomb-dataset", "my-api".parse().unwrap());

    let tracer = init_tracer(&metadata).expect("failed to instantiate opentelemetry tracing");

    tracing_subscriber::registry()
        .with(tracing_subscriber::EnvFilter::from_default_env())
        .with(tracing_opentelemetry::layer().with_tracer(tracer))
        .with(tracing_subscriber::fmt::layer())
        .try_init()
        .expect("failed to register tracer with registry");

    let tracer = global::tracer("ex.com/basic");

but I am getting:

2022-11-02T17:01:01.088429Z DEBUG hyper::client::connect::http: connecting to 52.5.162.226:443
2022-11-02T17:01:01.170767Z DEBUG hyper::client::connect::http: connected to 52.5.162.226:443
2022-11-02T17:01:01.171870Z DEBUG rustls::client::hs: No cached session for DnsName(DnsName(DnsName("api.honeycomb.io")))
2022-11-02T17:01:01.172555Z DEBUG rustls::client::hs: Not resuming any session
2022-11-02T17:01:01.269218Z DEBUG rustls::client::hs: ALPN protocol is Some(b"h2")
2022-11-02T17:01:01.269398Z DEBUG rustls::client::hs: Using ciphersuite TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
2022-11-02T17:01:01.269504Z DEBUG rustls::client::tls12::server_hello: Server supports tickets
2022-11-02T17:01:01.269766Z DEBUG rustls::client::tls12: ECDHE curve is ECParameters { curve_type: NamedCurve, named_group: secp256r1 }
2022-11-02T17:01:01.269843Z DEBUG rustls::client::tls12: Server DNS name is DnsName(DnsName(DnsName("api.honeycomb.io")))
2022-11-02T17:01:01.271123Z  WARN rustls::conn: Sending fatal alert BadCertificate
2022-11-02T17:01:01.271861Z DEBUG tonic::transport::service::reconnect: reconnect::poll_ready: hyper::Error(Connect, Custom { kind: InvalidData, error: InvalidCertificateData("invalid peer certificate: UnknownIssuer") })
2022-11-02T17:01:01.271967Z DEBUG tower::buffer::worker: service.ready=true processing request
2022-11-02T17:01:01.272169Z DEBUG tonic::transport::service::reconnect: error: error trying to connect: invalid peer certificate contents: invalid peer certificate: UnknownIssuer
OpenTelemetry trace error occurred. Exporter otlp encountered the following error(s): the grpc server returns error (The service is currently unavailable): , detailed error message: error trying to connect: invalid peer certificate contents: invalid peer certificate: UnknownIssuer

which seems to indicate that something about my TLS setup is not correct... Does anyone have a snippet of opentelemetry code in Rust that works with Honeycomb?

For reference, a complete solution to Rust/Honeycomb/Tracing/OpenTelemetry implementation is now available here: https://github.com/Dhghomon/rust_opentelemetry_honeycomb.

It does not need to use certs.

UPDATE: Expanding a little since this is the accepted answer now.

The core tracer initialization code is as follows:

fn init_tracer() -> Result<sdktrace::Tracer, TraceError> {
    opentelemetry_otlp::new_pipeline()
        .tracing()
        .with_exporter(
            opentelemetry_otlp::new_exporter()
                .http()
                .with_endpoint("https://api.honeycomb.io/v1/traces")
                .with_http_client(reqwest::Client::default())
                .with_headers(HashMap::from([
                    ("x-honeycomb-dataset".into(), DATASET_NAME.into()),
                    ("x-honeycomb-team".into(), API_KEY.into()),
                ]))
                .with_timeout(std::time::Duration::from_secs(2)),
        ) // Replace with runtime::Tokio if using async main
        .install_batch(opentelemetry::runtime::TokioCurrentThread)
}

The subscriber is configured like this:

let tracer = init_tracer().unwrap();
let telemetry = tracing_opentelemetry::layer().with_tracer(tracer);
let subscriber = tracing_subscriber::Registry::default().with(telemetry);
tracing::subscriber::set_global_default(subscriber).unwrap();