kubernetesopensearchjaegerobservability

Jaeger UI giving 400 BadRequest erorr

I am using Jaeger with OpenSearch as backend storage and am able to run the pod successfully in the Kubernetes Cluster but getting the below error in Jaeger UI.

I see some errors in Jaeger Query pod below

$ kubectl logs jaeger-query-869657fbcb-v6tlr -n telemetry
{"level":"info","ts":1705518488.5592744,"caller":"healthcheck/handler.go:99","msg":"Health Check server started","http-port":16687,"status":"unavailable"}
{"level":"info","ts":1705518488.5769525,"caller":"query/main.go:188","msg":"Archive storage not created","reason":"Archive storage not supported"}
{"level":"info","ts":1705518488.7728596,"caller":"query/main.go:136","msg":"Registering metrics handler with HTTP server","route":"/metrics"}
{"level":"info","ts":1705518488.772938,"caller":"healthcheck/handler.go:133","msg":"Health Check state change","status":"ready"}
{"level":"info","ts":1705518488.7729561,"caller":"query/main.go:145","msg":"Starting HTTP server","port":16686}
{"level":"error","ts":1705519069.3422198,"caller":"app/handler.go:462","msg":"HTTP handler, Internal Server Error","error":"Search service failed: elastic: Error 400 (Bad Request)","errorVerbose":"elastic: Error 400 (Bad Request)\nSearch service failed\ngithub.com/jaegertracing/jaeger/plugin/storage/es/spanstore.(*ServiceOperationStorage).getServices\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/plugin/storage/es/spanstore/service_operation.go:95\ngithub.com/jaegertracing/jaeger/plugin/storage/es/spanstore.(*SpanReader).GetServices\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/plugin/storage/es/spanstore/reader.go:207\ngithub.com/jaegertracing/jaeger/storage/spanstore/metrics.(*ReadMetricsDecorator).GetServices\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/storage/spanstore/metrics/decorator.go:102\ngithub.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).getServices\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/query/app/handler.go:157\ngithub.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).getServices-fm\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/query/app/handler.go:127\nnet/http.HandlerFunc.ServeHTTP\n\t/home/travis/.gimme/versions/go1.11.1.linux.amd64/src/net/http/server.go:1964\nnet/http.Handler.ServeHTTP-fm\n\t/home/travis/.gimme/versions/go1.11.1.linux.amd64/src/net/http/h2_bundle.go:5592\ngithub.com/jaegertracing/jaeger/vendor/github.com/opentracing-contrib/go-stdlib/nethttp.MiddlewareFunc.func4\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/opentracing-contrib/go-stdlib/nethttp/server.go:128\nnet/http.HandlerFunc.ServeHTTP\n\t/home/travis/.gimme/versions/go1.11.1.linux.amd64/src/net/http/server.go:1964\nnet/http.Handler.ServeHTTP-fm\n\t/home/travis/.gimme/versions/go1.11.1.linux.amd64/src/net/http/h2_bundle.go:5592\nnet/http.HandlerFunc.ServeHTTP\n\t/home/travis/.gimme/versions/go1.11.1.linux.amd64/src/net/http/server.go:1964\ngithub.com/jaegertracing/jaeger/vendor/github.com/gorilla/mux.(*Router).ServeHTTP\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/gorilla/mux/mux.go:114\ngithub.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers.CompressHandlerLevel.func1\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers/compress.go:146\nnet/http.HandlerFunc.ServeHTTP\n\t/home/travis/.gimme/versions/go1.11.1.linux.amd64/src/net/http/server.go:1964\ngithub.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers.recoveryHandler.ServeHTTP\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers/recovery.go:78\ngithub.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers.(*recoveryHandler).ServeHTTP\n\t<autogenerated>:1\nnet/http.serverHandler.ServeHTTP\n\t/home/travis/.gimme/versions/go1.11.1.linux.amd64/src/net/http/server.go:2741\nnet/http.(*conn).serve\n\t/home/travis/.gimme/versions/go1.11.1.linux.amd64/src/net/http/server.go:1847\nruntime.goexit\n\t/home/travis/.gimme/versions/go1.11.1.linux.amd64/src/runtime/asm_amd64.s:1333","stacktrace":"github.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).handleError\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/query/app/handler.go:462\ngithub.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).getServices\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/query/app/handler.go:158\ngithub.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).getServices-fm\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/query/app/handler.go:127\nnet/http.HandlerFunc.ServeHTTP\n\t/home/travis/.gimme/versions/go1.11.1.linux.amd64/src/net/http/server.go:1964\nnet/http.Handler.ServeHTTP-fm\n\t/home/travis/.gimme/versions/go1.11.1.linux.amd64/src/net/http/h2_bundle.go:5592\ngithub.com/jaegertracing/jaeger/vendor/github.com/opentracing-contrib/go-stdlib/nethttp.MiddlewareFunc.func4\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/opentracing-contrib/go-stdlib/nethttp/server.go:128\nnet/http.HandlerFunc.ServeHTTP\n\t/home/travis/.gimme/versions/go1.11.1.linux.amd64/src/net/http/server.go:1964\nnet/http.Handler.ServeHTTP-fm\n\t/home/travis/.gimme/versions/go1.11.1.linux.amd64/src/net/http/h2_bundle.go:5592\nnet/http.HandlerFunc.ServeHTTP\n\t/home/travis/.gimme/versions/go1.11.1.linux.amd64/src/net/http/server.go:1964\ngithub.com/jaegertracing/jaeger/vendor/github.com/gorilla/mux.(*Router).ServeHTTP\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/gorilla/mux/mux.go:114\ngithub.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers.CompressHandlerLevel.func1\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers/compress.go:146\nnet/http.HandlerFunc.ServeHTTP\n\t/home/travis/.gimme/versions/go1.11.1.linux.amd64/src/net/http/server.go:1964\ngithub.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers.recoveryHandler.ServeHTTP\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers/recovery.go:78\nnet/http.serverHandler.ServeHTTP\n\t/home/travis/.gimme/versions/go1.11.1.linux.amd64/src/net/http/server.go:2741\nnet/http.(*conn).serve\n\t/home/travis/.gimme/versions/go1.11.1.linux.amd64/src/net/http/server.go:1847"}
{"level":"error","ts":1705519069.3452828,"caller":"app/handler.go:462","msg":"HTTP handler, Internal Server Error","error":"Search service failed: elastic: Error 400 (Bad Request)","errorVerbose":"elastic: Error 400 (Bad Request)\nSearch service failed\ngithub.com/jaegertracing/jaeger/plugin/storage/es/spanstore.(*ServiceOperationStorage).getOperations\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/plugin/storage/es/spanstore/service_operation.go:127\ngithub.com/jaegertracing/jaeger/plugin/storage/es/spanstore.(*SpanReader).GetOperations\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/plugin/storage/es/spanstore/reader.go:216\ngithub.com/jaegertracing/jaeger/storage/spanstore/metrics.(*ReadMetricsDecorator).GetOperations\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/storage/spanstore/metrics/decorator.go:110\ngithub.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).getOperationsLegacy\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/query/app/handler.go:172\ngithub.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).getOperationsLegacy-fm\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/query/app/handler.go:131\nnet/http.HandlerFunc.ServeHTTP\n\t/home/travis/.gimme/versions/go1.11.1.linux.amd64/src/net/http/server.go:1964\nnet/http.Handler.ServeHTTP-fm\n\t/home/travis/.gimme/versions/go1.11.1.linux.amd64/src/net/http/h2_bundle.go:5592\ngithub.com/jaegertracing/jaeger/vendor/github.com/opentracing-contrib/go-stdlib/nethttp.MiddlewareFunc.func4\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/opentracing-contrib/go-stdlib/nethttp/server.go:128\nnet/http.HandlerFunc.ServeHTTP\n\t/home/travis/.gimme/versions/go1.11.1.linux.amd64/src/net/http/server.go:1964\nnet/http.Handler.ServeHTTP-fm\n\t/home/travis/.gimme/versions/go1.11.1.linux.amd64/src/net/http/h2_bundle.go:5592\nnet/http.HandlerFunc.ServeHTTP\n\t/home/travis/.gimme/versions/go1.11.1.linux.amd64/src/net/http/server.go:1964\ngithub.com/jaegertracing/jaeger/vendor/github.com/gorilla/mux.(*Router).ServeHTTP\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/gorilla/mux/mux.go:114\ngithub.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers.CompressHandlerLevel.func1\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers/compress.go:146\nnet/http.HandlerFunc.ServeHTTP\n\t/home/travis/.gimme/versions/go1.11.1.linux.amd64/src/net/http/server.go:1964\ngithub.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers.recoveryHandler.ServeHTTP\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers/recovery.go:78\ngithub.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers.(*recoveryHandler).ServeHTTP\n\t<autogenerated>:1\nnet/http.serverHandler.ServeHTTP\n\t/home/travis/.gimme/versions/go1.11.1.linux.amd64/src/net/http/server.go:2741\nnet/http.(*conn).serve\n\t/home/travis/.gimme/versions/go1.11.1.linux.amd64/src/net/http/server.go:1847\nruntime.goexit\n\t/home/travis/.gimme/versions/go1.11.1.linux.amd64/src/runtime/asm_amd64.s:1333","stacktrace":"github.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).handleError\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/query/app/handler.go:462\ngithub.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).getOperationsLegacy\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/query/app/handler.go:173\ngithub.com/jaegertracing/jaeger/cmd/query/app.(*APIHandler).getOperationsLegacy-fm\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/query/app/handler.go:131\nnet/http.HandlerFunc.ServeHTTP\n\t/home/travis/.gimme/versions/go1.11.1.linux.amd64/src/net/http/server.go:1964\nnet/http.Handler.ServeHTTP-fm\n\t/home/travis/.gimme/versions/go1.11.1.linux.amd64/src/net/http/h2_bundle.go:5592\ngithub.com/jaegertracing/jaeger/vendor/github.com/opentracing-contrib/go-stdlib/nethttp.MiddlewareFunc.func4\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/opentracing-contrib/go-stdlib/nethttp/server.go:128\nnet/http.HandlerFunc.ServeHTTP\n\t/home/travis/.gimme/versions/go1.11.1.linux.amd64/src/net/http/server.go:1964\nnet/http.Handler.ServeHTTP-fm\n\t/home/travis/.gimme/versions/go1.11.1.linux.amd64/src/net/http/h2_bundle.go:5592\nnet/http.HandlerFunc.ServeHTTP\n\t/home/travis/.gimme/versions/go1.11.1.linux.amd64/src/net/http/server.go:1964\ngithub.com/jaegertracing/jaeger/vendor/github.com/gorilla/mux.(*Router).ServeHTTP\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/gorilla/mux/mux.go:114\ngithub.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers.CompressHandlerLevel.func1\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers/compress.go:146\nnet/http.HandlerFunc.ServeHTTP\n\t/home/travis/.gimme/versions/go1.11.1.linux.amd64/src/net/http/server.go:1964\ngithub.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers.recoveryHandler.ServeHTTP\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/gorilla/handlers/recovery.go:78\nnet/http.serverHandler.ServeHTTP\n\t/home/travis/.gimme/versions/go1.11.1.linux.amd64/src/net/http/server.go:2741\nnet/http.(*conn).serve\n\t/home/travis/.gimme/versions/go1.11.1.linux.amd64/src/net/http/server.go:1847"}

enter image description here

Below is the manifest for Jaeger Config map and Jaeger Query

Jaeger Configmap

apiVersion: v1
kind: ConfigMap
metadata:
  name: jaeger-configuration
  namespace: telemetry
  labels:
    app: jaeger
    app.kubernetes.io/name: jaeger
data:
  span-storage-type: elasticsearch
  collector: |
    es:
      server-urls: http://opensearch:9200
      username: elastic
      password: changeme
    collector:
      zipkin:
        http-port: 9411
  query: |
    es:
      server-urls: http://opensearch:9200
      username: elastic
      password: changeme
  agent: |
    collector:
      host-port: "jaeger-collector:14267"

Jaeger Query


apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: jaeger-query
    namespace: telemetry
    labels:
      app: jaeger
      app.kubernetes.io/name: jaeger
      app.kubernetes.io/component: query
    annotations:
      eks.amazonaws.com/fargate-profile: jaeger-query
  spec:
    replicas: 1
    selector:
      matchLabels:
        app: jaeger
        app.kubernetes.io/name: jaeger
        app.kubernetes.io/component: query
    strategy:
      type: Recreate
    template:
      metadata:
        labels:
          app: jaeger
          app.kubernetes.io/name: jaeger
          app.kubernetes.io/component: query
        annotations:
          prometheus.io/scrape: "true"
          prometheus.io/port: "16686"
      spec:
        tolerations:
        - key: "eks.amazonaws.com/compute-type"
          operator: "Equal"
          value: "fargate"
          effect: "NoSchedule"
        containers:
        - image: jaegertracing/jaeger-query:1.9.0
          name: jaeger-query
          args: ["--config-file=/conf/query.yaml","--es.max-span-age=168h0m0s"]
          ports:
          - containerPort: 16686
            protocol: TCP
          readinessProbe:
            httpGet:
              path: "/"
              port: 16687
          volumeMounts:
          - name: jaeger-configuration-volume
            mountPath: /conf
          env:
          - name: SPAN_STORAGE_TYPE
            valueFrom:
              configMapKeyRef:
                name: jaeger-configuration
                key: span-storage-type
        volumes:
          - configMap:
              name: jaeger-configuration
              items:
                - key: query
                  path: query.yaml
            name: jaeger-configuration-volume

Jaeger Query Service

apiVersion: v1
  kind: Service
  metadata:
    name: jaeger-query
    namespace: telemetry
    labels:
      app: jaeger
      app.kubernetes.io/name: jaeger
      app.kubernetes.io/component: query
  spec:
    ports:
    - name: jaeger-query
      port: 16686
      protocol: TCP
      targetPort: 16686
    selector:
      app.kubernetes.io/name: jaeger
      app.kubernetes.io/component: query
    type: NodePort
Solution

  • This was resolved after using the latest image of Jaeger in the declarative syntax.