[SOLVED] Multithreaded algorithm where worker threads can schedule work

Multithreaded algorithm where worker threads can schedule work

I'm trying to implement a parallel algorithm in Rust using Rayon.

It seems to work fine if you can divide up the work ahead of time and assign it to threads. The problem is that I want the worker threads themselves to be able to schedule work and this I'm finding to be much more difficult. I'd love to have some thread safe queue perhaps which I can let the threads add work items to. And perhaps then the main thread could poll this queue and assign it to threads.

Any ideas on how to implement something like this? I'm using Rayon now, but if some other threading crate would make this easier I'd be open to changing.

Solution

I ended up solving this with rayon::join

My algorithm is recursive, and works by subdividing the x,y plane into four quadrants, recursively. Inside the code where I have to spawn subtasks, it looks like this.

// Split the current rectangle up into four rectangles and process them individually.

let sub_partitions = subdivide_partition(p);

assert_eq!(sub_partitions.len(), 4);

rayon::join(|| process_partition(image_info, &sub_partitions[0], pixels),
            || process_partition(image_info, &sub_partitions[1], pixels));
rayon::join(|| process_partition(image_info, &sub_partitions[2], pixels),
            || process_partition(image_info, &sub_partitions[3], pixels));

NOTE: I'm not super familiar with rayon, and the original example code I used to get my project started used rayon::scope, so I was a bit surprised that no scope was needed in order to use join. In fact the only rayon code in my whole codebase is the code shown above, which is quite convenient.