I know there are libraries for tracing requests in distributed systems based on OpenTracing and OpenTelemetry; these all work because the requests are connected/chained(microservices talking to each other). How to trace when systems/services are disconnected?
Service X calls Service Y. Y performs some processing then writes to a shared location . A component Z polls shared location(after hrs) and does processing.
I want to know if for each of the request generated by X, the status at each of these 3 systems.
A simple way to track this would be have a table and update DB Entries. However considering the fact that I might need to retry certain things, I had thought of introducing a workflow engine(Cadence) that opens up a workflow at Service X. Then at each of the Services the workflow is updated. If Request Reaches Y, a step in w/f is completed. If it does not reach Z the workflow is left open and after some time I could generate an Email etc. saying the workflow failed.
I need help in understanding if this is the right way and/or different methods to do this.
This is one of the most typical use cases of Cadence Workflow. The recommended solution is let service X to start a workflow. It will execute an activity to make a service call to service Y. If possible, let Component Z to send a signal to the workflow, telling that the processing is done. Or you can have another activity to keep polling the status.
Note that back off retry can be done easily for activities supported by Cadence.
Then use a timer in the workflow to trigger an activity to send email.
Finally the status of this workflow is still as a local variable of the workflow. You can implement a Query handler in the workflow to get status. Or you can implement a background activity to report the progress proactively .
Open tracing library framework is for solving another problem. It is to measuring and analyzing service dependency and latency. In fact, Cadence workflow also supports open tracing here is the example. It is for service level, not for a particular requests or processing.