I have a few Java applications that have working otel tracing implementations. The details of how this works is largely irrelevant (implemented using micrometer and spring-boot-actuator) but the point is when I point these programs directly at jaeger:4318, the traces show up as expected. Quite a few of my traces are not necessary so I wanted to remove certain unhelpful traces or ideally rename them. This lead me to discovering the otel-collector which seemed like the ideal thing to take traces in from my service and send them out to jaeger but trying to get them working has completely failed, there is no error just nothing happens.
Here is my api service's docker image
api-service:
image: <api-image>
restart: on-failure
depends_on:
- mysql
- zookeeper
- redis
- kafka
- user-app
- ff4j-ui
volumes:
- $PWD/data/config:/config
environment:
SPRING_REDIS_HOST: <host>
AWS_ACCESS_KEY_ID: <AWS_ACCESS_KEY_ID>
SPRING_DATASOURCE_PASSWORD: <PASSWORD>
AWS_SECRET_ACCESS_KEY: <AWS_SECRET_ACCESS_KEY>
CORS_EXPOSED_HEADERS: <HEADERS>
XSRF_DOMAIN: 127.0.0.1
ZOOKEEPER_CONNECTION: zookeeper:2181
JAEGER_SAMPLER_TYPE: const
SPRING_DATASOURCE_USERNAME: <USERNAME>
REDIS_SESSION_HOST: redis
JDBC_INCLUDES: QUERY, KEYS, FETCH
OTEL_EXPORTER_OTLP_ENDPOINT=: http://0.0.0.0:4318
CCS_SERVICE_NAME: <SERVICE-NAME>
FEATUREFLAG_HOSTURL: http://ff4j-ui:8099
CORS_ALLOWED_ORIGINS: http://127.0.0.1:8081
SWAGGER_ENABLED: 'false'
CORS_ALLOWED_METHODS: GET,POST,HEAD,OPTIONS,PATCH,PUT,DELETE
JAEGER_SAMPLER_PARAM: '1'
JAEGER_AGENT_HOST: jaeger
HTTP_SERVICE_CONNECT_TIMEOUT: '99999'
HTTP_SERVICE_READ_TIMEOUT: '999999'
SPRING_DATASOURCE_URL: <DATASOURCE-URL>
SPRING_KAFKA_BOOTSTRAPSERVERS: kafka:9092
MANAGEMENT_OTLP_TRACING_ENDPOINT: http://0.0.0.0:4318
ports:
- 8082:8080
- 7988:5005
Here is the otel-collector's docker file
otel-collector:
image: otel/opentelemetry-collector-contrib:latest
depends_on:
- jaeger
volumes:
- $PWD/data/config/otel-collector-config.yml:/etc/otel-collector-config.yml
ports:
- 4317:4317
- 4318:4318
- 1888:1888
- 13133:13133
- 55679:55679
Here is the jaeger docker file:
image: jaegertracing/all-in-one:latest # At least 1.35 if you want to have enabled collector
container_name: jaeger
environment:
COLLECTOR_ZIPKIN_HOST_PORT: 9411
COLLECTOR_OTLP_ENABLED: true
SPAN_STORAGE_TYPE: memory
LOG_LEVEL: debug
JAEGER_LOG_LEVEL: debug
ports:
- "16686:16686"
- "16685:16685"
And here is the config file for the otel-collector
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
filter:
traces:
exclude:
spans:
- 'name == "security filterchain before"'
- 'name == "authorize request"'
exporters:
otlp/jaeger:
endpoint: jaeger:4317
tls:
insecure: true # Use this for local Docker environments without TLS
debug:
verbosity: detailed
service:
telemetry:
logs:
level: debug
extensions: [zpages]
pipelines:
traces:
receivers: [otlp]
exporters: [otlp/jaeger,debug]
When I set it to export to debug it this works and displays logs in the console but sending to jaeger does not send anywhere and does not show up at http://localhost:16686/ even though when I set my api-service's MANAGEMENT_OTLP_TRACING_ENDPOINT to jaeger:4318 it works flawlessly.
I have tried everything, removing all processors from the config file so I know that's not the issue, changing it to point at jaeger:4318 or jaeger:14250, trying newer versions of jaeger I've even messed around with setting the MANAGEMENT_OTLP_TRACING_ENDPOINT to otel-collector:4138 or otel-collector:4137 (which isn't really necessary given the traces show up in debug mode).
Regardless of what I do, no traces end up going out from my collector to jaeger. Either I would like to fix this or failing that just find another solution to do what I want to do.
My issue was simply that I was using a program to compile all of my docker-compose files into 1. This program only kept the "essential" parts and didn't keep the command: --config /etc/otel/config.yaml part of my otel-collector so the config wasn't being loaded properly into the collector.