My REST service is performing lots of http queries which I would like to monitor in my dashboard. In it's APM view I can choose from servlet.request operation name or netty.client.request. The former one is the primary operation, while the latter one is additional operation.
What I managed to do was that I am able to build a widget in the dashboard which uses metric-type source like this: default_zero(sum:trace.netty.client.request.hits.by_http_status{$env,$service,!http.status_code:200}.as_count()) and it works, it shows me the number of errors client is getting, but when I click on the widget and choose show related traces I do not see traces related with netty.client.request operation, but the ones related with servlet.request. The reason for it was that it generates following query env:prod service:payments-braintree-gateway -@http.status_code:200, which does not include operation name, nor "all spans" (next paragraph explains it)
So I went to traces view in APM and I tried doing the query I wanted, there. Initially I was only able to choose one operation in the filters - servlet.request, but I noticed a button next to search bar labeled in, where you can choose in which set of spans to search for. There are 2 options: service entry spans and all... Switching to all made it possible to create a query I am interested in: $env $service operation_name:netty.client.request -@http.status_code:(200) and it works... So the only thing left was to use it in the dashboard.
Now when I am back in the dashboard's widget I fiddled with the query in json, I tried to edit context menu links, I even changed metric type from "metric" to APM... All in vein. Looks like it is impossible to use non service entry spans in widgets.
Funny thing is that I even exported from APM's service view error graph to dashboard and then clicked "show related traces" it still showed incorrect thing, while "show related traces" option in the graph in the APM takes me to the correct view in traces
So... is it possible to fix "show related traces" button in the DataDog dashboards?
Looks like the only doable solution is to hide standard traces link and create new custom link. Downside is that it is going to open a new APM browser tab instead showing traces in the modal at the right hand side of the screen, but at least it works. Still ideas how to tackle it differently are welcome of course :)