python-3.xgrafanaopen-telemetryopen-telemetry-collectorgrafana-tempo

Send traces directly from python application to Grafana Cloud's tempo instance without Grafana agent

Here is python script that sends test trace directly to tempo instance :

import base64
from typing import Tuple

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import \
    OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import Tracer, TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

def create_span_exporter_for_tempo() -> Tuple[Tracer, BatchSpanProcessor]:

    resource = Resource(attributes={"service.name": constants.APP_NAME})

    headers = {
        "Authorization": f"Basic {base64.b64encode(f'{constants.GRAFANA_USERNAME}:{constants.GRAFANA_PASSWORD}'.encode('ascii')).decode('ascii')}"
    }
    tempo_exporter = OTLPSpanExporter(endpoint=constants.TEMPO_ENDPOINT, headers=headers)

    trace.set_tracer_provider(TracerProvider(resource=resource))
    tracer = trace.get_tracer(__name__)
    span_processor = BatchSpanProcessor(tempo_exporter)

    trace.get_tracer_provider().add_span_processor(span_processor)
    return tracer, span_processor

tracer, span_processor = create_span_exporter_for_tempo()
with tracer.start_as_current_span("test") as span:
        print("test!")
        span.set_attribute("check", "works")

My Grafana Cloud setup looks like this : enter image description here

Here are some of variations of tempo endpoint tried :

1. https://tempo-prod-04-prod-us-east-0.grafana.net/tempo
2. https://tempo-prod-04-prod-us-east-0.grafana.net/tempo/v1/traces
3. https://tempo-prod-04-prod-us-east-0.grafana.net/v1/traces
4. https://tempo-prod-04-prod-us-east-0.grafana.net:443/tempo/v1/traces
5. https://tempo-prod-04-prod-us-east-0.grafana.net/tempo/v1/trace

All of these endpoint return the same output :

Failed to export batch code: 404, reason: 404 page not found

enter image description here

Please note that, I have tested out this tempo instance by setting up grafana agent & I was able to send traces via grafana agent.

My requirement is to send traces without deploying grafana agent as I am sending traces from lambda & do not want to run agent somewhere else 24X7.

Here is python library being used

Solution

  • You have to use OTLP Gateway. Make note of the following values that you’ll need in order to write OLTP:

    Doc: https://grafana.com/docs/grafana-cloud/data-configuration/otlp/send-data-otlp/