I have a Datadog APM trace from my Rails monolith that looks a bit like this:
Sidekiq job
Each line is a separate span. I would like to search for the failed HTTP requests and then look up certain tags from the corresponding Sidekiq span, but I don't see a way to do so? I am approaching this by querying Datadog APM for something like
service:my_service resource_name:POST @peer.hostname=my.other.service.host @http.path_group:/route status:error
but that, predictably, gives me back the Faraday traces, and if I open any of them I can visually inspect them and see their Sidekiq parent, but I can't programatically pull tags from that parents span into my Datadog facets or columns.
It's probably not useful to you now, but at the time this question was asked, that was not possible.
Now, however, DataDog does have a "Traces" tab where you can search for traces based on tags on separate spans. This new feature is called Trace Queries: https://docs.datadoghq.com/tracing/trace_explorer/trace_queries/
With Trace Queries, you can find entire traces based on the properties of multiple spans and the relationships between those spans within the structure of the trace. To create a trace query, you define two or more span queries and then specify the relationship within the searched-for trace structure of the spans that are returned by each span query.
Note: It only searches based on traces indexed from DataDog's intelligent indexing, not any of your custom retention rules, so what traces you can search from is much more limited.