I have been trying to elegantly handle the case where an asynchronous worker thread produces both a result and (possibly) identifies more work that needs to be done. To think of it another way, if you are traversing a tree and doing work at each node, the worker is processing a node in the middle of the tree and has discovered child nodes that require work themselves.
Is it a reasonable design for the worker thread to add more jobs to the job queue? This would require the worker to know something about the concurrency system of which it is a part, which seems to violate some unwritten rule for me. How else has this problem been tackled?
Is it a reasonable design for the worker thread to add more jobs to the job queue?
I've certainly written code like this before where the worker task has access to the task queue and can add additional tasks to it. But that said, I understand your reticence from a design standpoint.
How else has this problem been tackled?
One alternative would be for the the results from the task to contain a collection of additional tasks to be added to the list. So when the thread which is reaping the finished tasks looks at the results, it would add any additional tasks to the work queue at that point.
This does mean that the reaper will need to process the results in real time for optimal concurrency.