I am using the following crates: tracing, opentelemetry, tracing-opentelemetry, opentelemetry-application-insights, but the console is flooded with DEBUG/TRACE messages now (related to connection with application insights). I have 3 questions:
Here is my main:
async fn init_tracing() {
let app_insights_connection_string = "...";
let client = reqwest::Client::new();
let tracer = opentelemetry_application_insights::new_pipeline_from_connection_string(
app_insights_connection_string,
)
.expect("valid connection string")
.with_client(client)
.with_live_metrics(true)
.with_service_name("Rust Web App")
.with_sample_rate(0.3)
.install_batch(opentelemetry_sdk::runtime::Tokio);
let telemetry = OpenTelemetryLayer::new(tracer);
let subscriber = tracing_subscriber::registry()
.with(tracing_subscriber::fmt::layer())
.with(telemetry);
subscriber.init();
}
#[tokio::main]
async fn main() {
init_tracing().await;
...
let app = Router::new()
.merge(open_api_router())
.nest_service("/api", api_router(app_state))
.layer(CompressionLayer::new())
.layer(TraceLayer::new_for_http());
...
opentelemetry::global::shutdown_tracer_provider();
}
Here is my Cargo.toml:
...
# HTTP Client
reqwest = { version = "0.11.27", features = ["default", "json", "blocking", "rustls-tls"] }
# Tracing
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["default", "env-filter", "serde", "chrono", "tracing", "serde_json"] }
tracing-opentelemetry = { version = "0.23.0", features = ["metrics", "async-trait", "thiserror"] }
opentelemetry = "0.22.0"
opentelemetry_sdk = { version = "0.22.1", features = ["default", "tokio", "serde", "serde_json", "opentelemetry-http", "rt-tokio", "metrics", "http", "trace"] }
opentelemetry-http = { version = "0.11.1", features = ["tokio", "reqwest"] }
opentelemetry-application-insights = { version = "0.30.0", features = ["reqwest", "reqwest-client", "reqwest-blocking-client", "metrics", "live-metrics", "reqwest-client-rustls", "reqwest-blocking-client-rustls", "reqwest-client-vendored-tls"] }
...
Here is a section of the run console output:
2024-04-27T18:43:08.789483Z DEBUG establish{db.system="postgresql" otel.kind="client"}: diesel_tracing::pg: establishing postgresql connection
2024-04-27T18:43:08.809463Z DEBUG establish{db.system="postgresql" otel.kind="client"}: diesel_tracing::pg: querying postgresql connection information
2024-04-27T18:43:08.810534Z INFO uni_hub_api: ✅ Connection to the data is successful!
2024-04-27T18:43:08.810704Z DEBUG establish{db.system="postgresql" otel.kind="client"}: diesel_tracing::pg: establishing postgresql connection
2024-04-27T18:43:08.811979Z INFO uni_hub_api: ⬆️ Migrations were applied successfully!
2024-04-27T18:43:08.813812Z INFO uni_hub_api: 🚀 Server started successfully! Listening on 0.0.0.0:8080...
2024-04-27T18:43:08.830483Z DEBUG establish{db.system="postgresql" otel.kind="client"}: diesel_tracing::pg: querying postgresql connection information
2024-04-27T18:43:13.803307Z TRACE hyper::client::pool: checkout waiting for idle connection: ("https", switzerlandnorth.livediagnostics.monitor.azure.com)
2024-04-27T18:43:13.803684Z TRACE hyper::client::pool: checkout waiting for idle connection: ("https", switzerlandnorth-0.in.applicationinsights.azure.com)
2024-04-27T18:43:13.804410Z DEBUG reqwest::connect: starting new connection: https://switzerlandnorth-0.in.applicationinsights.azure.com/
2024-04-27T18:43:13.804409Z DEBUG reqwest::connect: starting new connection: https://switzerlandnorth.livediagnostics.monitor.azure.com/
2024-04-27T18:43:13.805209Z TRACE hyper::client::connect::http: Http::connect; scheme=Some("https"), host=Some("switzerlandnorth.livediagnostics.monitor.azure.com"), port=None
2024-04-27T18:43:13.805223Z TRACE hyper::client::connect::http: Http::connect; scheme=Some("https"), host=Some("switzerlandnorth-0.in.applicationinsights.azure.com"), port=None
2024-04-27T18:43:13.805851Z DEBUG hyper::client::connect::dns: resolving host="switzerlandnorth-0.in.applicationinsights.azure.com"
2024-04-27T18:43:13.805880Z DEBUG hyper::client::connect::dns: resolving host="switzerlandnorth.livediagnostics.monitor.azure.com"
2024-04-27T18:43:14.067899Z DEBUG hyper::client::connect::http: connecting to 51.107.52.200:443
2024-04-27T18:43:14.099041Z DEBUG hyper::client::connect::http: connected to 51.107.52.200:443
2024-04-27T18:43:14.149468Z DEBUG hyper::client::connect::http: connecting to 51.107.48.68:443
2024-04-27T18:43:14.175926Z TRACE hyper::client::conn: client handshake Http1
2024-04-27T18:43:14.176562Z TRACE hyper::client::client: handshake complete, spawning background dispatcher task
2024-04-27T18:43:14.178442Z TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Init, writing: Init, keep_alive: Busy }
2024-04-27T18:43:14.178763Z TRACE hyper::client::pool: checkout dropped for ("https", switzerlandnorth.livediagnostics.monitor.azure.com)
2024-04-27T18:43:14.179401Z TRACE encode_headers: hyper::proto::h1::role: Client::encode method=POST, body=Some(Known(184))
2024-04-27T18:43:14.180248Z TRACE hyper::proto::h1::encode: sized write, len = 184
2024-04-27T18:43:14.180483Z TRACE hyper::proto::h1::io: buffer.flatten self.len=489 buf.len=184
2024-04-27T18:43:14.180743Z DEBUG hyper::proto::h1::io: flushed 673 bytes
2024-04-27T18:43:14.180937Z TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Init, writing: KeepAlive, keep_alive: Busy }
2024-04-27T18:43:14.181274Z DEBUG hyper::client::connect::http: connected to 51.107.48.68:443
2024-04-27T18:43:14.210945Z TRACE hyper::proto::h1::conn: Conn::read_head
2024-04-27T18:43:14.211246Z TRACE hyper::proto::h1::io: received 367 bytes
2024-04-27T18:43:14.211496Z TRACE parse_headers: hyper::proto::h1::role: Response.parse bytes=367
2024-04-27T18:43:14.212061Z TRACE parse_headers: hyper::proto::h1::role: Response.parse Complete(367)
2024-04-27T18:43:14.212935Z DEBUG hyper::proto::h1::io: parsed 9 headers
2024-04-27T18:43:14.213157Z DEBUG hyper::proto::h1::conn: incoming body is empty
2024-04-27T18:43:14.213403Z TRACE hyper::proto::h1::conn: maybe_notify; read_from_io blocked
2024-04-27T18:43:14.213651Z TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Init, writing: Init, keep_alive: Idle }
2024-04-27T18:43:14.213864Z TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Init, writing: Init, keep_alive: Idle }
2024-04-27T18:43:14.214743Z TRACE hyper::client::pool: put; add idle connection for ("https", switzerlandnorth.livediagnostics.monitor.azure.com)
2024-04-27T18:43:14.214995Z DEBUG hyper::client::pool: pooling idle connection for ("https", switzerlandnorth.livediagnostics.monitor.azure.com)
2024-04-27T18:43:14.215486Z TRACE hyper::client::pool: idle interval checking for expired
2024-04-27T18:43:14.249996Z TRACE hyper::client::conn: client handshake Http1
2024-04-27T18:43:14.250311Z TRACE hyper::client::client: handshake complete, spawning background dispatcher task
2024-04-27T18:43:14.250622Z TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Init, writing: Init, keep_alive: Busy }
2024-04-27T18:43:14.250926Z TRACE hyper::client::pool: checkout dropped for ("https", switzerlandnorth-0.in.applicationinsights.azure.com)
2024-04-27T18:43:14.251396Z TRACE encode_headers: hyper::proto::h1::role: Client::encode method=POST, body=Some(Known(1057))
2024-04-27T18:43:14.251705Z TRACE hyper::proto::h1::encode: sized write, len = 1057
2024-04-27T18:43:14.251865Z TRACE hyper::proto::h1::io: buffer.flatten self.len=177 buf.len=1057
2024-04-27T18:43:14.252136Z DEBUG hyper::proto::h1::io: flushed 1234 bytes
2024-04-27T18:43:14.252312Z TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Init, writing: KeepAlive, keep_alive: Busy }
2024-04-27T18:43:14.284064Z TRACE hyper::proto::h1::conn: Conn::read_head
2024-04-27T18:43:14.284317Z TRACE hyper::proto::h1::io: received 308 bytes
2024-04-27T18:43:14.284499Z TRACE parse_headers: hyper::proto::h1::role: Response.parse bytes=308
2024-04-27T18:43:14.285003Z TRACE parse_headers: hyper::proto::h1::role: Response.parse Complete(240)
2024-04-27T18:43:14.285279Z DEBUG hyper::proto::h1::io: parsed 6 headers
2024-04-27T18:43:14.285482Z DEBUG hyper::proto::h1::conn: incoming body is chunked encoding
2024-04-27T18:43:14.285658Z TRACE hyper::proto::h1::decode: decode; state=Chunked { state: Start, chunk_len: 0, extensions_cnt: 0 }
2024-04-27T18:43:14.285859Z TRACE hyper::proto::h1::decode: Read chunk start
2024-04-27T18:43:14.285993Z TRACE hyper::proto::h1::decode: Read chunk hex size
2024-04-27T18:43:14.286149Z TRACE hyper::proto::h1::decode: Read chunk hex size
2024-04-27T18:43:14.286268Z TRACE hyper::proto::h1::decode: Chunk size is 62
2024-04-27T18:43:14.286421Z DEBUG hyper::proto::h1::decode: incoming chunked header: 0x3E (62 bytes)
2024-04-27T18:43:14.286577Z TRACE hyper::proto::h1::decode: Chunked read, remaining=62
2024-04-27T18:43:14.286718Z TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Body(Chunked { state: BodyCr, chunk_len: 0, extensions_cnt: 0 }), writing: KeepAlive, keep_alive: Busy }
2024-04-27T18:43:14.286991Z TRACE hyper::proto::h1::decode: decode; state=Chunked { state: BodyCr, chunk_len: 0, extensions_cnt: 0 }
2024-04-27T18:43:14.287150Z TRACE hyper::proto::h1::decode: Read chunk hex size
2024-04-27T18:43:14.287278Z TRACE hyper::proto::h1::io: received 5 bytes
2024-04-27T18:43:14.287395Z TRACE hyper::proto::h1::decode: Read chunk hex size
2024-04-27T18:43:14.287509Z TRACE hyper::proto::h1::decode: Chunk size is 0
2024-04-27T18:43:14.287624Z TRACE hyper::proto::h1::decode: end of chunked
2024-04-27T18:43:14.287744Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-04-27T18:43:14.287923Z TRACE hyper::proto::h1::conn: maybe_notify; read_from_io blocked
2024-04-27T18:43:14.288070Z TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Init, writing: Init, keep_alive: Idle }
2024-04-27T18:43:14.288223Z TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Init, writing: Init, keep_alive: Idle }
2024-04-27T18:43:14.288366Z TRACE hyper::client::pool: put; add idle connection for ("https", switzerlandnorth-0.in.applicationinsights.azure.com)
2024-04-27T18:43:14.288507Z DEBUG hyper::client::pool: pooling idle connection for ("https", switzerlandnorth-0.in.applicationinsights.azure.com)
2024-04-27T18:43:14.288715Z TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Init, writing: Init, keep_alive: Idle }
2024-04-27T18:43:18.797873Z TRACE hyper::client::pool: take? ("https", switzerlandnorth-0.in.applicationinsights.azure.com): expiration = Some(90s)
2024-04-27T18:43:18.798350Z DEBUG hyper::client::pool: reuse idle connection for ("https", switzerlandnorth-0.in.applicationinsights.azure.com)
2024-04-27T18:43:18.798871Z TRACE encode_headers: hyper::proto::h1::role: Client::encode method=POST, body=Some(Known(933))
2024-04-27T18:43:18.799331Z TRACE hyper::proto::h1::encode: sized write, len = 933
2024-04-27T18:43:18.799635Z TRACE hyper::proto::h1::io: buffer.flatten self.len=176 buf.len=933
2024-04-27T18:43:18.800016Z DEBUG hyper::proto::h1::io: flushed 1109 bytes
2024-04-27T18:43:18.800317Z TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Init, writing: KeepAlive, keep_alive: Busy }
2024-04-27T18:43:18.833117Z TRACE hyper::proto::h1::conn: Conn::read_head
2024-04-27T18:43:18.833488Z TRACE hyper::proto::h1::io: received 310 bytes
2024-04-27T18:43:18.833801Z TRACE parse_headers: hyper::proto::h1::role: Response.parse bytes=310
2024-04-27T18:43:18.834195Z TRACE parse_headers: hyper::proto::h1::role: Response.parse Complete(240)
2024-04-27T18:43:18.834651Z DEBUG hyper::proto::h1::io: parsed 6 headers
2024-04-27T18:43:18.834946Z DEBUG hyper::proto::h1::conn: incoming body is chunked encoding
2024-04-27T18:43:18.835206Z TRACE hyper::proto::h1::decode: decode; state=Chunked { state: Start, chunk_len: 0, extensions_cnt: 0 }
2024-04-27T18:43:18.835485Z TRACE hyper::proto::h1::decode: Read chunk start
2024-04-27T18:43:18.835777Z TRACE hyper::proto::h1::decode: Read chunk hex size
2024-04-27T18:43:18.835959Z TRACE hyper::proto::h1::decode: Read chunk hex size
2024-04-27T18:43:18.836138Z TRACE hyper::proto::h1::decode: Chunk size is 64
2024-04-27T18:43:18.836293Z DEBUG hyper::proto::h1::decode: incoming chunked header: 0x40 (64 bytes)
2024-04-27T18:43:18.836485Z TRACE hyper::proto::h1::decode: Chunked read, remaining=64
2024-04-27T18:43:18.836663Z TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Body(Chunked { state: BodyCr, chunk_len: 0, extensions_cnt: 0 }), writing: KeepAlive, keep_alive: Busy }
2024-04-27T18:43:18.836995Z TRACE hyper::proto::h1::decode: decode; state=Chunked { state: BodyCr, chunk_len: 0, extensions_cnt: 0 }
2024-04-27T18:43:18.837136Z TRACE hyper::proto::h1::decode: Read chunk hex size
2024-04-27T18:43:18.837296Z TRACE hyper::proto::h1::io: received 5 bytes
2024-04-27T18:43:18.837449Z TRACE hyper::proto::h1::decode: Read chunk hex size
2024-04-27T18:43:18.837581Z TRACE hyper::proto::h1::decode: Chunk size is 0
2024-04-27T18:43:18.837716Z TRACE hyper::proto::h1::decode: end of chunked
2024-04-27T18:43:18.837835Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-04-27T18:43:18.837975Z TRACE hyper::proto::h1::conn: maybe_notify; read_from_io blocked
2024-04-27T18:43:18.838124Z TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Init, writing: Init, keep_alive: Idle }
2024-04-27T18:43:18.838256Z TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Init, writing: Init, keep_alive: Idle }
2024-04-27T18:43:18.838408Z TRACE hyper::client::pool: put; add idle connection for ("https", switzerlandnorth-0.in.applicationinsights.azure.com)
2024-04-27T18:43:18.838571Z DEBUG hyper::client::pool: pooling idle connection for ("https", switzerlandnorth-0.in.applicationinsights.azure.com)
2024-04-27T18:43:18.838788Z TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Init, writing: Init, keep_alive: Idle }
2024-04-27T18:43:19.224346Z TRACE hyper::client::pool: take? ("https", switzerlandnorth.livediagnostics.monitor.azure.com): expiration = Some(90s)
2024-04-27T18:43:19.224570Z DEBUG hyper::client::pool: reuse idle connection for ("https", switzerlandnorth.livediagnostics.monitor.azure.com)
2024-04-27T18:43:19.224982Z TRACE encode_headers: hyper::proto::h1::role: Client::encode method=POST, body=Some(Known(184))
2024-04-27T18:43:19.225308Z TRACE hyper::proto::h1::encode: sized write, len = 184
2024-04-27T18:43:19.225582Z TRACE hyper::proto::h1::io: buffer.flatten self.len=489 buf.len=184
2024-04-27T18:43:19.225936Z DEBUG hyper::proto::h1::io: flushed 673 bytes
2024-04-27T18:43:19.226238Z TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Init, writing: KeepAlive, keep_alive: Busy }
2024-04-27T18:43:19.256445Z TRACE hyper::proto::h1::conn: Conn::read_head
2024-04-27T18:43:19.256798Z TRACE hyper::proto::h1::io: received 367 bytes
2024-04-27T18:43:19.257123Z TRACE parse_headers: hyper::proto::h1::role: Response.parse bytes=367
2024-04-27T18:43:19.257475Z TRACE parse_headers: hyper::proto::h1::role: Response.parse Complete(367)
2024-04-27T18:43:19.258103Z DEBUG hyper::proto::h1::io: parsed 9 headers
2024-04-27T18:43:19.258393Z DEBUG hyper::proto::h1::conn: incoming body is empty
2024-04-27T18:43:19.258633Z TRACE hyper::proto::h1::conn: maybe_notify; read_from_io blocked
2024-04-27T18:43:19.258974Z TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Init, writing: Init, keep_alive: Idle }
2024-04-27T18:43:19.259218Z TRACE hyper::proto::h1::conn: flushed({role=client}): State { reading: Init, writing: Init, keep_alive: Idle }
2024-04-27T18:43:19.259470Z TRACE hyper::client::pool: put; add idle connection for ("https", switzerlandnorth.livediagnostics.monitor.azure.com)
2024-04-27T18:43:19.259642Z DEBUG hyper::client::pool: pooling idle connection for ("https", switzerlandnorth.livediagnostics.monitor.azure.com)
In Application Insights, in Live Metrics, the metrics are shown correctly but there is no sample telemetry. It just says Not supported by this Agent/SDK version.
The DEBUG/TRACE logs are also sent to application insights and are then available in Logs -> traces query.
I would be so grateful if someone can help me with this. I am new to Rust and I've spent the last 2 days looking into this.
So far, I've tried to add .with_max_level on the tracing layers, but I get this error:
the method `with_max_level` exists for struct `OpenTelemetryLayer<_, Tracer>`, but its trait bounds were not satisfied [E0599] method cannot be called on `OpenTelemetryLayer<_, Tracer>` due to unsatisfied trait bounds Note: the following trait bounds were not satisfied: `OpenTelemetryLayer<_, opentelemetry_sdk::trace::Tracer>: MakeWriter<'_>` which is required by `OpenTelemetryLayer<_, opentelemetry_sdk::trace::Tracer>: MakeWriterExt<'_>` `&OpenTelemetryLayer<_, opentelemetry_sdk::trace::Tracer>: MakeWriter<'_>` which is required by `&OpenTelemetryLayer<_, opentelemetry_sdk::trace::Tracer>: MakeWriterExt<'_>` `&mut OpenTelemetryLayer<_, opentelemetry_sdk::trace::Tracer>: MakeWriter<'_>` which is required by `&mut OpenTelemetryLayer<_, opentelemetry_sdk::trace::Tracer>: MakeWriterExt<'_>`
How to hide the DEBUG/TRACE logs from console and application insights?
tracing-subscriber, you can use its EnvFilter feature to set the log level. Check the below steps to set the log level using init_tracing function.use tracing_subscriber::EnvFilter;
async fn init_tracing() {
let app_insights_connection_string = "...";
let client = reqwest::Client::new();
let tracer = opentelemetry_application_insights::new_pipeline_from_connection_string(
app_insights_connection_string,
)
.expect("valid connection string")
.with_client(client)
.with_live_metrics(true)
.with_service_name("Rust Web App")
.with_sample_rate(0.3)
.install_batch(opentelemetry_sdk::runtime::Tokio);
let telemetry = OpenTelemetryLayer::new(tracer);
let filter = EnvFilter::from_default_env()
.add_directive(tracing_subscriber::filter::LevelFilter::INFO.into());
let subscriber = tracing_subscriber::registry()
.with(tracing_subscriber::fmt::layer())
.with(telemetry)
.with(filter);
subscriber.init();
}
This above setup will hide DEBUG and TRACE logs from both the console and Application Insights
init_tracing() in your application's entry point Call the init_tracing() function at the start of the application. Adjust the configuration parameters such as service name and sample rate according to the requirement.Console logs:
INFO uni_hub_api: ✅ Connection to the data is successful!
INFO uni_hub_api: ⬆️ Migrations were applied successfully!
INFO uni_hub_api: 🚀 Server started successfully! Listening on 0.0.0.0:8080...
Reference: