datadog

Datadog APM: Search tags in multiple spans of the same trace


I have a Datadog APM trace from my Rails monolith that looks a bit like this:

Sidekiq job

Each line is a separate span. I would like to search for the failed HTTP requests and then look up certain tags from the corresponding Sidekiq span, but I don't see a way to do so? I am approaching this by querying Datadog APM for something like

service:my_service resource_name:POST @peer.hostname=my.other.service.host @http.path_group:/route status:error

but that, predictably, gives me back the Faraday traces, and if I open any of them I can visually inspect them and see their Sidekiq parent, but I can't programatically pull tags from that parents span into my Datadog facets or columns.


Solution

  • It's probably not useful to you now, but at the time this question was asked, that was not possible.

    Now, however, DataDog does have a "Traces" tab where you can search for traces based on tags on separate spans. This new feature is called Trace Queries: https://docs.datadoghq.com/tracing/trace_explorer/trace_queries/

    With Trace Queries, you can find entire traces based on the properties of multiple spans and the relationships between those spans within the structure of the trace. To create a trace query, you define two or more span queries and then specify the relationship within the searched-for trace structure of the spans that are returned by each span query.

    Image of trace queries example from datadog documentation

    Note: It only searches based on traces indexed from DataDog's intelligent indexing, not any of your custom retention rules, so what traces you can search from is much more limited.